trjhtr

I've been experimenting with scikit-learn for the past few months and have been finding it difficult to move away from the inbuilt datasets (such as Twenty Newsgroups and Iris) and onto my own text datasets.

I have finally managed to get something working, but am keen to get my code sense-checked just in case I'm tricking myself into thinking I'm doing better than I am.

The following code is based on this Sklearn tutorial, but uses my own dataset of approximately 25,000 text files spread across 273 subdirectories in the main project folder. Each directory name serves as a descriptive label for the text files contained within it.

The objectives of the following code are as follows:

1. Iterate over each subdirectory path in the main project folder to extract the name of each of label (these are appended to the categories list, which is passed in to sklearn's load_files() function.


2. Use train_test_split() to hold out 40% of the dataset as test data


3. Transform the training and testing data to tf-idf
   


4. Train the classifier


5. Evaluate the classifier's predictions with the test data

The code below works and I'm currently averaging around 0.7 accuracy (so obviously, there's some improvement still needed).

The main areas I feel might be lacking are the way in which I'm bringing the names of the categories in and the way I'm dealing with the test/train split.

Some feedback from more seasoned developers would be gratefully received.

The code

import sklearn
import numpy as np
from glob import glob
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import SGDClassifier
from sklearn import metrics
from sklearn.pipeline import Pipeline

# Get paths to labelled data
rawFolderPaths = glob("/Users/danielhoadley/PycharmProjects/trainer/!labelled_data_reportXML/*/")


categories = 

# Extract the folder paths, reduce down to the label and append to the categories list
for i in rawFolderPaths:
 string1 = i.replace('/Users/danielhoadley/PycharmProjects/trainer/!labelled_data_reportXML/','')
 category = string1.strip('/')
 categories.append(category)

# Load the data
print ('nLoading the dataset...n')
docs_to_train = sklearn.datasets.load_files("/Users/danielhoadley/PycharmProjects/trainer/!labelled_data_reportXML",
 description=None, categories=categories, load_content=True,
 encoding='utf-8', shuffle=True, random_state=42)

# Split the dataset into training and testing sets
print ('nBuilding out hold-out test sample...n')
X_train, X_test, y_train, y_test = train_test_split(docs_to_train.data, docs_to_train.target, test_size=0.4)

# THE TRAINING DATA

# Transform the training data into tfidf vectors
print ('nTransforming the training data...n')
count_vect = CountVectorizer(stop_words='english')
X_train_counts = count_vect.fit_transform(raw_documents=X_train)

tfidf_transformer = TfidfTransformer(use_idf=True)
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

# THE TEST DATA

# Transform the test data into tfidf vectors
print ('nTransforming the test data...n')
count_vect = CountVectorizer(stop_words='english')
X_test_counts = count_vect.fit_transform(raw_documents=X_test)

tfidf_transformer = TfidfTransformer(use_idf=True)
X_test_tfidf = tfidf_transformer.fit_transform(X_test_counts)
print (X_test_tfidf.shape)

docs_test = X_test

# Construct the classifier pipeline 
text_clf = Pipeline([('vect', CountVectorizer(stop_words='english')),
 ('tfidf', TfidfTransformer(use_idf=True)),
 ('clf', SGDClassifier(loss='hinge', penalty='l2',
 alpha=1e-3, random_state=42, verbose=1)),
])

# Fit the model to the training data
text_clf.fit(X_train, y_train)

# Run the test data into the model
predicted = text_clf.predict(docs_test)

# Calculate mean accuracy of predictions
print (np.mean(predicted == y_test))

# Generate labelled performance metrics
print(metrics.classification_report(y_test, predicted,
 target_names=docs_to_train.target_names))