Datumbox Machine Studying Framework model 0.8.0 launched

Datumbox Framework v0.8.0 is out and packs a number of highly effective options! This model brings new Preprocessing, Characteristic Choice and Mannequin Choice algorithms, new highly effective Storage Engines that give higher management on how the Fashions and the Dataframes are saved/loaded, a number of pre-trained Machine Studying fashions and plenty of reminiscence & pace enhancements. Obtain it now from Github or Maven Central Repository.

One of many foremost targets of model 0.8.0 was to enhance the Storage mechanisms of the framework and make disk-based coaching accessible to all of the supported algorithms. The brand new storage engines give higher management over how and when the fashions are being continued. One necessary change is that the fashions aren’t being saved robotically after the match() technique is completed however as a substitute one must explicitly name the save() technique offering the title of the mannequin. This allows us not solely to discard simpler short-term algorithms with out going via a serialization section but in addition to save lots of/load the Dataframes:

Configuration configuration = Configuration.getConfiguration();
Dataframe knowledge = ...; //load a dataframe right here

MaximumEntropy.TrainingParameters params = new MaximumEntropy.TrainingParameters();
MaximumEntropy mannequin = MLBuilder.create(params, getConfiguration());
mannequin.match(knowledge);
mannequin.save("MyModel"); //save the mannequin utilizing the precise title
mannequin.shut();

knowledge.save("MyData"); //save the info utilizing a particular title
knowledge.shut();

knowledge = Dataframe.Builder.load("MyData", configuration); //load the info
mannequin = MLBuilder.load(MaximumEntropy.class, "MyModel", configuration); //load the mannequin
mannequin.predict(knowledge);
mannequin.delete(); //delete the mannequin

At present we help two storage engines: The InMemory engine which may be very quick because it hundreds all the things in reminiscence and the MapDB engine which is slower however permits disk-based coaching. You possibly can management which engine you utilize by altering your datumbox.configuration.properties or you’ll be able to programmatically modify the configuration objects. Every engine has its personal configuration file however once more you’ll be able to modify all the things programmatically:

Configuration configuration = Configuration.getConfiguration(); //conf from properties file

configuration.setStorageConfiguration(new InMemoryConfiguration()); //use In-Reminiscence engine
//configuration.setStorageConfiguration(new MapDBConfiguration()); //use MapDB engine

Please word that in each engines, there’s a listing setting which controls the place the fashions are being saved (inMemoryConfiguration.listing and mapDBConfiguration.listing properties in config information). Be sure you change them or else the fashions will probably be written on the short-term folder of your system. For extra data on the way you construction the configuration information take a look on the Code Instance mission.

With the brand new Storage mechanism in place, it’s now possible to share publicly pre-trained fashions that cowl the areas of Sentiment Evaluation, Spam Detection, Language Detection, Matter Classification and all the opposite fashions which are accessible through the Datumbox API. Now you can obtain and use all of the pre-trained fashions in your mission with out requiring calling the API and with out being restricted by the variety of each day calls. At present the revealed fashions are educated utilizing the InMemory storage engine and so they help solely English. On future releases, I plan to offer help for extra languages.

Within the new framework, there are a number of modifications on the general public strategies of lots of the lessons (therefore it isn’t backwards suitable). Essentially the most notable distinction is on the best way the fashions are initialized. As we noticed within the earlier code instance, the fashions aren’t immediately instantiated however as a substitute the MLBuilder class is used to both create or load a mannequin. The coaching parameters are supplied on to the builder and so they can’t be modified with a setter.

One other enchancment is on the best way we carry out Mannequin Choice. The v0.8.0 introduces the brand new modelselection bundle which gives all the required instruments for validating and measuring the efficiency of our fashions. Within the metrics subpackage we offer a very powerful validation metrics for classification, clustering, regression and suggestion. Notice that the ValidationMetrics are faraway from every particular person algorithm and they’re not saved along with the mannequin. The framework gives the brand new splitters subpackage which allows splitting the unique dataset utilizing totally different schemes. At present Okay-fold splits are carried out utilizing the KFoldSplitter class whereas partitioning the dataset right into a coaching and check set will be achieved with the ShuffleSplitter. Lastly to rapidly validate a mannequin, the framework gives the Validator class. Right here is how one can carry out Okay-fold cross validation inside a few traces of code:

ClassificationMetrics vm = new Validator<>(ClassificationMetrics.class, configuration)
    .validate(new KFoldSplitter(ok).break up(knowledge), new MaximumEntropy.TrainingParameters());

The brand new Preprocessing bundle replaces the previous Knowledge Transformers and provides higher management on how we scale and encode the info earlier than the machine studying algorithms. The next algorithms are supported for scaling numerical variables: MinMaxScaler, StandardScaler, MaxAbsScaler and BinaryScaler. For encoding categorical variables into booleans you need to use the next strategies: OneHotEncoder and CornerConstraintsEncoder. Right here is how you need to use the brand new algorithms:

StandardScaler numericalScaler = MLBuilder.create(
    new StandardScaler.TrainingParameters(), 
    configuration
);
numericalScaler.fit_transform(trainingData);

CornerConstraintsEncoder categoricalEncoder = MLBuilder.create(
    new CornerConstraintsEncoder.TrainingParameters(), 
    configuration
);
categoricalEncoder.fit_transform(trainingData);

One other necessary replace is the truth that the Characteristic Choice bundle was rewritten. At present all function choice algorithms concentrate on particular datatypes, making it attainable to chain totally different strategies collectively. In consequence the TextClassifier and the Modeler lessons obtain a listing of function selector parameters moderately than only one.

As talked about earlier all of the algorithms now help disk-based coaching, together with those who use Matrices (solely exception is the Help Vector Machines). The brand new storage engine mechanism even makes it attainable to configure some algorithms or dataframes to be saved in reminiscence whereas others on disk. A number of pace enhancements had been launched primarily as a result of new storage engine mechanism but in addition as a result of tuning of particular person algorithms akin to those within the DPMM household.

Final however not least the brand new model updates all of the dependencies to their newest variations and removes a few of them such because the the commons-lang and lp_solve. The commons-lang, which was used for HTML parsing, is changed with a quicker customized HTMLParser implementation. The lp_solve is changed with a pure Java simplex solver which signifies that Datumbox not requires particular system libraries put in on the working system. Furthermore lp_solve needed to go as a result of it makes use of LGPLv2 which isn’t suitable with the Apache 2.0 license.

The model 0.8.0 brings a number of extra new options and enhancements on the framework. For an in depth view of the modifications please examine the Changelog.

Don’t neglect to clone the code of Datumbox Framework v0.8.0 from Github, try the Code Examples and obtain the pre-trained Machine Studying fashions from Datumbox Zoo. I’m wanting ahead to your feedback and options.