trjhtr

I am trying to improve the run time of a program I wrote in R. Generally, what I am doing is feeding a function a data frame of values and generating a prediction off of operations on specific columns. The function is a custom function that is being used with sapply (code below). What I'm doing is much too large to provide any meaningful example, so instead I will try to describe the inputs to the process. I know this will restrict how helpful answers can be, but I am interested in any ideas for optimizing the time it takes me to compute a prediction. Currently it is taking me about 10 seconds to generate one prediction (running the sapply for one line of a dataframe).

mean_rating <- function(df)
 user<-df$user
 movie<-df$movie
 u_row<-which(U_lookup == user)[1]
 m_row<-which(M_lookup==movie)[1]

 knn_match<- knn_txt[u_row,1:100]

 knn_match1<-as.numeric(unlist(knn_match))

 dfm_test<- dfm[knn_match1,]

 dfm_mov<- dfm_test[,m_row] # row number from DFM associated with the query_movie
 C<-mean(dfm_mov)



test<-sapply(1:nrow(probe_test),function(x) mean_rating(probe_test[x,]))

Inputs: dfm is my main data matrix, users in the rows and movies in the columns. Very sparse.

> str(dfm)
Formal class 'dgTMatrix' [package "Matrix"] with 6 slots
 ..@ i : int [1:99072112] 378 1137 1755 1893 2359 3156 3423 4380 5103 6762 ...
 ..@ j : int [1:99072112] 0 0 0 0 0 0 0 0 0 0 ...
 ..@ Dim : int [1:2] 480189 17770
 ..@ Dimnames:List of 2
 .. ..$ : NULL
 .. ..$ : NULL
 ..@ x : num [1:99072112] 4 5 4 1 4 5 4 5 3 3 ...
 ..@ factors : list()

probe_test is my test set, the set I'm trying to predict for. The actual probe test contains approximately 1.4 million rows but I am trying it on a subset first to optimize the time. It is being fed into my function.

> str(probe_test)
'data.frame': 6 obs. of 6 variables:
 $ X : int 1 2 3 4 5 6
 $ movie : int 1 1 1 1 1 1
 $ user : int 1027056 1059319 1149588 1283744 1394012 1406595
 $ Rating : int 3 3 4 3 5 4
 $ Rating_Date: Factor w/ 1929 levels "2000-01-06","2000-01-08",..: 1901 1847 1911 1312 1917 1803
 $ Indicator : int 1 1 1 1 1 1

U_lookup is the lookup I use to convert between user id and the line of the matrix a user is in since we lose user id's when they are converted to a sparse matrix.

> str(U_lookup)
'data.frame': 480189 obs. of 1 variable:
 $ x: int 10 100000 1000004 1000027 1000033 1000035 1000038 1000051 1000053 1000057 ...

M_lookup is the lookup I use to convert between movie id and the column of a matrix a movie is in for similar reasons as above.

> str(M_lookup)
'data.frame': 17770 obs. of 1 variable:
 $ x: int 1 10 100 1000 10000 10001 10002 10003 10004 10005 ...

knn_text contains the 100 nearest neighbors for all the lines of dfm

> str(knn_txt)
'data.frame': 480189 obs. of 200 variables:

Does anyone have suggestions on how I could improve performance within R? Does anyone have other language suggestions? I am slightly familiar with Python so I've been looking into that one, but if anyone has specific tips on redoing this in Python I would be grateful as I'm very inexperienced with it.

Without exact your data I could think of some improvements.
Trying to avoid redundant operations.

# order data.frame by users and movies
probe_test <- probe_test[with(probe_test, order(user, move)), ]
# initialize resulting column
probe_test$res <- rep(as.numeric(NA), nrow(probe_test))

knn_txt_red <- knn_txt[, 1:100] # reduce outside of the loop
for (user in unique(probe_test$user)) # for each unique user
 u_row <- which(U_lookup == user)[1] # get your id
 knn_match <- knn_txt_red[u_row, ]
 knn_match1 <- as.numeric(unlist(knn_match))
 userI <- probe_test$user == user
 movies <- probe_test$movie[userI] #get all user movies
 m_row <- which(M_lookup %in% movies) # get indexes
 dfm_mov <- dfm[knn_match1, m_row] #select all cols of those movies for user
 x <- colMeans(dfm_mov) # calculate mean for each row
 probe_test[userI, 'res'] <- x # add the results to data.frame

As I do not have your data, there are probably/maybe some errors in code.

There are probably better ways to do this, but as I mentioned, it is hard to think of any without any example data.