See in Action¶

Let’s see how TrendyPy works with a few use cases.

Stock Data¶

In this demo, I’d like to show you how to use TrendyPy in some stock data between 2018-01-01 and 2020-06-28. You can download the data from here to reproduce the demo.

Let’s say we have some stock data from a combination of tech and banking. And, we want to identify an unknown trend if it’s a tech stock or banking. For this purpose, we’ll use FB (i.e. Facebook), GOOGL (i.e. Google), AMZN (i.e Amazon), BAC (i.e. Bank of America) and WFC (i.e. Wells Fargo) for training data then AAPL (i.e. Apple) and c (i.e. Citigroup) for prediction data.

But first, here is how the data looks.

In [1]: import pandas as pd

In [2]: import matplotlib.pyplot as plt

In [3]: df = pd.read_csv('stock_data.csv')

In [4]: df.plot()
Out[4]: <matplotlib.axes._subplots.AxesSubplot at 0x7f195b408810>

If we cluster like this, the expensive stocks like GOOGL and AMZN will alone constitute one cluster which it’s clearly not intended. So, let’s scale first.

In [5]: from trendypy import utils

In [6]: df = df.apply(utils.scale_01)

In [7]: df.plot()
Out[7]: <matplotlib.axes._subplots.AxesSubplot at 0x7f195b407bd0>

It’s a bit apparent that BAC, WFC and c are different than the others. Let’s put sectors side by side to see the difference better.

In [8]: fig, axes_ = plt.subplots(nrows=1, ncols=2)

In [9]: axes_[0].set_title('Tech')
Out[9]: Text(0.5, 1.0, 'Tech')

In [10]: axes_[1].set_title('Banking')
Out[10]: Text(0.5, 1.0, 'Banking')

In [11]: df[['AAPL', 'FB', 'GOOGL', 'AMZN']].plot(ax=axes_[0])
Out[11]: <matplotlib.axes._subplots.AxesSubplot at 0x7f19611fe9d0>

In [12]: df[['BAC', 'WFC', 'c']].plot(ax=axes_[1])
Out[12]: <matplotlib.axes._subplots.AxesSubplot at 0x7f1959ae6810>

Now, we can use the training data to fit. Remember, we’re setting AAPL and c aside to predict later and only fit by using the rest.

In [13]: from trendypy.trendy import Trendy

In [14]: trendy = Trendy(n_clusters=2) # 2 for tech and banking

In [15]: trendy.fit([df.FB, df.GOOGL, df.AMZN, df.BAC, df.WFC])

In [16]: trendy.labels_
Out[16]: [0, 0, 0, 1, 1]

You can also use fit_predict method for this purpose, it’s essentially the same.

In [17]: trendy.fit_predict([df.FB, df.GOOGL, df.AMZN, df.BAC, df.WFC])
Out[17]: [0, 0, 0, 1, 1]

As expected, it successfully assigns FB, GOOGL and AMZN into the first cluster (i.e. 0) and BAC and WFC into the second (i.e. 1). So, we can name 0 as tech and 1 as banking.

Now, let’s make predictions on the prediction data that we set aside earlier (i.e. AAPL, c).

In [18]: trendy.predict([df.AAPL]) # expecting `0` since AAPL is a part of tech
Out[18]: [0]

In [19]: trendy.predict([df.c]) # expecting `1` since c is a part of banking
Out[19]: [1]

As seen above, it correctly predicts trends.

You can easily pickle the model object to be used later with to_pickle method.

In [20]: trendy.to_pickle('my_first_trendy.pkl')

And, that’s all.

Image Clustering¶

If you have the proper distance metric function for the right data, you can use TrendyPy to even cluster images. In this demo, I’ll use black & white images from MPEG7 CE Shape-1 Part B database. The goal is to correctly cluster the images and assign new ones to the appropriate clusters. Here are some simple images that’ll be used to create the clusters. Each image is slightly different than the others in the same group. You can download the images if you want to reproduce the demo.

car-01.gif¶	car-02.gif¶	car-03.gif¶
carriage-02.gif¶	carriage-03.gif¶	carriage-04.gif¶
chopper-01.gif¶	chopper-02.gif¶	chopper-03.gif¶

Define a function to read the image and convert to a numpy array.

In [21]: from PIL import Image

In [22]: import numpy as np

In [23]: def load_image(file):
   ....:     img = Image.open(file)
   ....:     img.load()
   ....:     return np.asarray(img, dtype="int32")
   ....: 

Read images and assign them into lists.

In [24]: cars = [
   ....:     load_image('image_data/car-01.gif'),
   ....:     load_image('image_data/car-02.gif'),
   ....:     load_image('image_data/car-03.gif')
   ....: ]
   ....: 

In [25]: carriages = [
   ....:     load_image('image_data/carriage-02.gif'),
   ....:     load_image('image_data/carriage-03.gif'),
   ....:     load_image('image_data/carriage-04.gif')
   ....: ]
   ....: 

In [26]: choppers = [
   ....:     load_image('image_data/chopper-01.gif'),
   ....:     load_image('image_data/chopper-02.gif'),
   ....:     load_image('image_data/chopper-03.gif')
   ....: ]
   ....: 

Euclidean Distance can be used to calculate the similarity between images. So, let’s import euclidean_distance from utils module, then assign it as algorithm argument during the initialization.

In [27]: from trendypy.trendy import Trendy

In [28]: from trendypy.utils import euclidean_distance

In [29]: trendy = Trendy(n_clusters=3, algorithm=euclidean_distance)

In [30]: trendy.fit(cars + carriages + choppers)

In [31]: trendy.labels_
Out[31]: [0, 0, 0, 1, 1, 1, 2, 2, 2]

As expected, it correctly clusters these simple images. Let’s see if it predicts new data correctly.

../_images/carriage-20.gif — carriage-20.gif¶

../_images/chopper-08.gif — chopper-08.gif¶

In [32]: new_car = load_image('image_data/car-20.gif')

In [33]: new_carriage = load_image('image_data/carriage-20.gif')

In [34]: new_chopper = load_image('image_data/chopper-08.gif')

In [35]: trendy.predict([new_car, new_carriage, new_chopper])
Out[35]: [0, 1, 2]

Looks like it correctly predicts new data as well.

Note

Because of the limitation of the selected metric function (i.e. Euclidean Distance), I had to cherry pick images with exact same sizes (i.e. 352×288). Depending on the function you choose, you may or may not do the same.

Custom Metric¶

TrendyPy is flexible enough to be able to utilize user defined metrics. In this example, I’ll show how to create your own metric and use it during the clustering.

Let’s say we want to cluster DNA sequences and need a metric to do that. Needleman–Wunsch algorithm is an algorithm used in bioinformatics to align protein or nucleotide sequences. It’s not a metric but it inspires us to create our own metric. The metric basically compares two sequences with same length and it penalizes each mismatch by increasing the distance by p then divides it to total length.

In [36]: def my_metric(x, y, p=1):
   ....:     assert len(x) == len(y)
   ....:     dist = 0
   ....:     for i in range(len(x)):
   ....:         if x[i] != y[i]:
   ....:             dist += p
   ....:     return dist/len(x)
   ....: 

As you can see, you just need to consider inputs and output of your custom function. Specifically,

Input must have x and y for two data points to compare. You may have other default arguments (e.g. p).
Output must be a float. 0 indicates same and greater is farther.

Note

Technically, any float range should work as the output of the custom function as long as greater is farther. However, it won’t be named as metric in that case.

Anyway, let’s use it.

In [37]: set_of_sequences = [
   ....:     'AAATTT', 'AAACTT', 'AAATCT', # group 1
   ....:     'GACTAG', 'GGCTAG', 'GACAAG' # group 2
   ....: ]
   ....: 

In [38]: from trendypy.trendy import Trendy

In [39]: trendy = Trendy(
   ....:     n_clusters=2, # there are 2 groups
   ....:     algorithm=my_metric # this is where to set custom metric
   ....: )
   ....: 

In [40]: trendy.fit(set_of_sequences)

In [41]: trendy.labels_
Out[41]: [0, 0, 0, 1, 1, 1]

It clearly clusters first and second group. Now, let’s see on new data.

In [42]: new_seq1 = 'AAAGGT' # similar to group 1

In [43]: new_seq2 = 'GTCCAG' # similar to group 2

In [44]: trendy.predict([new_seq1, new_seq2])
Out[44]: [0, 1]

Very simple.

TrendyPy API Reference