ive got a 6 gb csv file with the last 5000 days of price info for around 4800 companies merged with quarterly financial data like assets revenue etc merged with macro indicators like interest rates and all of that interpolated and cleaned up before passing to an lstm with many many layers with tuned hyper params. instead of using a standard loss value you should use mean absolute percentage error because if a stock is trading at 1$ and you have a loss of 1$ you would be off by 100% but if its trading at 1000$ and yoru loss is 10 then you would be off by 1%. also try to use time distributed lstm layers at the input and to use leaky relu not just relu so that parts of the model that arnt use often dont just become dead weight . Also use a deeper model to capture the complex relationship between the columns in the training data and MOST IMPORTANTLY..... Kill any over fitting with lots of dropout layers and batch norm

@CodeTradingCafe

5 ай бұрын

Thanks a lot for your input I will pin your comment for viewers building a LSTM system, will consider these ideas as well in the future. Thank you again.

@danielgoldney1151

3 ай бұрын

hey man, I'm doing a university project comparing deep learning methods of stock prediction vs traditional methods such as regression. Can I ask how accurate your model is and where you sourced the data from?

@gamingwithvillain1731

3 ай бұрын

can you give link to your code

@maciejkolibabski7491

3 ай бұрын

Bump, how accurate the model is ?

@sevi4481

2 ай бұрын

Can I buy the CSV file ?

The biggest pitfall of anyone starting with time series predictions (of any kind)! Predicting actual values instead of predicting the difference between values. Great video explaining why!

@CodeTradingCafe

Жыл бұрын

Thank you for your support, it actually is a widespread error all over the internet.

@qwertasd7

Жыл бұрын

to be usuable feeding it should at least be delta data based too, ea a hight of 101 next day 105 delta 4 ...vs 201 205, is also 4 the height itself doesnt contain information its not part of the pattern only a result, to create generic knowledge, its not height dependency but pattern dependency you want...

@eggmel1

Жыл бұрын

@@CodeTradingCafevery interesting video. I was wondering whether there is another one proposing ways to come up with a model that may exhibit some predictive value

@andresponce2497

10 ай бұрын

Using Bayesian Optimization I tried optimize hiperparameter such as layers, neurons, dropout, learning rate, backcandles another else but until now I can't predict the next close price, I use 1 minute candle but I can't get a good prediction, indeed I add unsupervised model before supervised model (lstm) and I can't improve the final model... any suggest about what I could do?

@Mammel248

10 ай бұрын

@@andresponce2497 the only way I have found to make my prediction better (not good, just a bit better) is to include sentiment analysis

Great video, excellent observation about the good predictions phenomena, almost 90% of articles and papers i came across online show high accuracy, that is not applicable for the future only for test data, which is useless!

@CodeTradingCafe

3 ай бұрын

Thank you! yes actually even academic papers are biased!

You just saved me a LOT of time, thank you 🙏🏼❤️

@CodeTradingCafe

10 ай бұрын

I am glad these videos are of help, thank you for your comment! Good luck!

That has really helped me! Straight to my notebooks for some experiments!!

@CodeTradingCafe

Жыл бұрын

Glad it's of use! Good luck! ... long coding hours ahead 😂

Thank you a lot for your effort and the valuable content of this channel :)

@CodeTradingCafe

Жыл бұрын

Thank you for your support, it's really great that the content is useful!

Great video! how would we modify this to predict multiple targets instead of one?

@CodeTradingCafe

10 ай бұрын

Hi, thank you, you can add a new category into the target column, but be aware that more categories might make it more difficult for the model to forecast all the possibilities... but that's the beauty of the challenge :)

For the bad predictions graph, I think it can be used as a golden crossover strategy between test and predicted prices

@CodeTradingCafe

Жыл бұрын

You might as well use a simple moving average probably the same outcome

Excellent video.

@CodeTradingCafe

Жыл бұрын

Thank you!

I tried your code for a few stocks and the results are amazing. Thank you

@CodeTradingCafe

Жыл бұрын

I am glad you liked it. Just warning if you are using this videos code it's NOT supposed to work well for real trading.

@publiccake

Ай бұрын

How do you pull data for stocks?

I Just got started with algo trading and I was wondering if using a LSTM model is really worth exploring due to this issue that has been brought up by you in this video. If not, do you have any suggestions on what model works best for forex trading in small timeframes such as 1min or 5 min charts? Great video by the way!

@CodeTradingCafe

Жыл бұрын

Thank you for your support! Actually I noticed that the more advanced/complicated methods work less in trading, surprisingly simple methods are still the most efficient, for example support resistance and candle patterns, what people tend to ignore is the trade management it's more important than your indicator/forecast.

Excellent explanation. Does your model predict different numbers after each running? Isn't it worth to make range of predictions like in Monte Carlo simulation?

@CodeTradingCafe

Жыл бұрын

Hi thank you, I don't think any neural net approach is worth it for trading predictions, at this point neural nets perform very poorly on this type of problems.

Great video and explanation! Thanks!

@CodeTradingCafe

Жыл бұрын

Nice emojis lol thank you

Nice ! But I sincerely think that data scaling is also a source of problems as most of the features are continuous (ATL, +inf). Therefore, usual scaling methods assume that the previous ATH will never be reached in the future in which case the model, no matter how sophisticated it is, will never predict a breakthrough (new ATHs).

@CodeTradingCafe

Жыл бұрын

Good observation ! I think these models can only be trained to read the current trend and they can be combined with other models/indicators

@kadourkadouri3505

Жыл бұрын

@@CodeTradingCafe Precisely! for your information ATH stands for All time high (the maximum historical price). Thanks for the content

@domenicperito4635

Жыл бұрын

scaled_value = (value - median) / IQR

@kadourkadouri3505

Жыл бұрын

@@domenicperito4635 more precisely please without abbreviations

@domenicperito4635

Жыл бұрын

@@kadourkadouri3505 To calculate the IQR, follow these steps: Arrange the data in ascending order. Find the median, or the middle value, of the dataset. If there is an even number of data points, the median is the average of the two middle values. Divide the dataset into two halves: the lower half (below the median) and the upper half (above the median). Find the first quartile (Q1), which is the median of the lower half of the data. Find the third quartile (Q3), which is the median of the upper half of the data. Calculate the IQR by subtracting Q1 from Q3: IQR = Q3 - Q1.

its an eye opening video for the algo traders . People should stop publishing fake articles on the ML

@CodeTradingCafe

Жыл бұрын

Yes ML techniques are still very weak for trading maybe just for trend detection.

Good point bro, instead of predicting some precise values, we can build some models to predict for example trends or convergence based on the right indicators and still alot of preprocessing techniques must be applied to reach lets say some reasonable results.

@CodeTradingCafe

Жыл бұрын

Actually I think the only info we can quickly get from lstm might be the current trend... I will see if I can make it for next week, lots of work 🙂

@lordofallworlds9244

Жыл бұрын

@@CodeTradingCafe which is still very beneficial toward price action strategies 👌🏽

@CodeTradingCafe

Жыл бұрын

Yes, I think it's the most challenging part of the algo ... Guessing the overall trend.

@lordofallworlds9244

Жыл бұрын

@@CodeTradingCafe The most challenging part in trading in general! And if you're a smart trader you know trying to predict prices is the worse thing you can do, but I am going to add this to my strategy in the hopes it will give me an edge anyway. It won't be profitable overnight but I'm sure it'll become more accurate after a few consistent months of fine tuning the model

@CodeTradingCafe

Жыл бұрын

I would be careful with this model as so far it's not really working, but other indicators we have seen on this channel can be really good mainly candle patterns and candle wick absence are the best I have found so far.

Thanks for the links bro 😊

@CodeTradingCafe

Жыл бұрын

Glad they're useful, good luck!

Great video! Have you ever thought about using a bot to make buy and sell decisions instead of predicting whether the ticket will go up or down? Using deep Q leaning techniques like DQN? Just a suggestion!

@CodeTradingCafe

Жыл бұрын

Thank you, I did think of reinforcement learning but it involves more parameters than expected sl tp trend trade volume ... Might be challenging but I admit I bam eager to try it 😂

@sagarraut0007

10 ай бұрын

@@CodeTradingCafe , Can you share a video on reinforcement learning or DQN? if you try it. It will really helpful :)

I tried to do exactly the same 2 years ago and I gave up. Congrats for the achievement!!

@CodeTradingCafe

11 ай бұрын

Thank you for your support, although it's not really working but at least we know :)

@kirannagaraj3364

10 ай бұрын

this is great. Were you able to repurpose LSTm to predict the trend?

yes delayed prediction still a problem when using LSTM model, maybe prediction based on market sentiment is more useful

@connectrRomania

Жыл бұрын

Also not useful cause you don't know the market sentiment in which period could affect the prices

@CodeTradingCafe

Жыл бұрын

@@connectrRomania exactly I mean we could easily scrap few articles per day and check the sentiment but when the effect will take place is another story... I would say weekly timeframe in this case is a good candidate but drowning in open trades fees is a problem.

THANCK YOU SO MUTCH IT WORKINR FIN

@CodeTradingCafe

7 ай бұрын

Thank you for your support.

can i use the trained model to trade on the pair that it wasnt trained on ?, ie trading on usdjpy using a model that was trained on usdchf

@CodeTradingCafe

Ай бұрын

Hi, I guess not, models barely work on the same currency and mostly they fail, I am not sure switching currencies would work.

Would love to see this built getting beginning to end and then modified and back tested. I was thinking, couldn't you see how high the green line was above the black line on average and add a line of code that was predicted close minus 20 pips equals new target?

@CodeTradingCafe

8 ай бұрын

Hi, thank you for your input. regarding your question can you please point out the video time related to this?

@BlackCatSyndicate

8 ай бұрын

i guess different parts in the video, but its related to the two line graph lines you display and how they are not exactly in sync with each other. If you knew how far the one was off from the other, you could just add a calculation to add X-amount of pips to the calculation to adjust the line relative to the other line. @@CodeTradingCafe

@CodeTradingCafe

8 ай бұрын

Ah ok now I see what you mean, actually no it doesn't work because the translation is on the x-axis and what we see on the y-axis is just the result of it, if we force it let's say 20 pips down it would only look nice for the eye but tricky because the difference is not always 20 pips it's just not seen from the scale of the presented figure.

thanks for the clarification - maybe worth removing the OHLC and add a lower EMA and then feed to the model? then the result may not be the same previous price?

@CodeTradingCafe

Жыл бұрын

Good idea actually using the ema instead of the candles here can be good. But we would still need the closing price data for comparison.

@rezasadeghi2520

Жыл бұрын

@@CodeTradingCafe thanks for your reply. I think the difference between close [i] and close [i-1] or return would be a better option compared to closing price? I guess if we have to then result can be converted to closing price if required. Another issue if there is a difference between the adj close and other OHLC price - i would suggest to convert the OHLC to adjusted values and run the model. it may not be the case for Forex but it is definitely the for stocks.

@eggmel1

Жыл бұрын

@@CodeTradingCafe also have you thought about removing the tgt variable from the "independent" set?

What the best books or course to get better in algo trading ?

@CodeTradingCafe

Жыл бұрын

There is no best book, straight answer is you have to master basics in python or any language, and then research strategies and try them out couple of years later you'll find that you can guess if a strategy is worth it before trying it. However I do have my plan that I follow when putting a new strategy together.

This is why i pushed off neural networks as not priority, you need to have analysis done before utilizing a neural network, its just a better eay of going thriugh your data but without intelligent analysis tactics its just gonna be random hit or miss

@CodeTradingCafe

Жыл бұрын

Yes exactly a random hit and miss with a straight approach like this. Nothing comes easy I guess :)

Sorry, I'm quite dumb. I'm trying to understand why for a simple classification model using the next closing price wouldn't work If I'm trying to catch a trend (and really don't care about how much will go up or down), looking at the predicted close isn't enough? So for example if I do: prev_actual_close and gather all results for the testing data, with (just for the example) 90% success rate, doesn't that mean I *could* use it as a "going go go up, not sure how much" type of flag? I know there are other (maybe) better classification models, but I'm just getting started and have much to learn still, your videos are helping a lot so thank you!

@CodeTradingCafe

11 ай бұрын

Hi not dumb at all, I did a video previously using machine learning just for classification trend up and trend down kzread.info/dash/bejne/o5Nk1NiwgMWvoco.html, it did work to an extent but not very robust in the sense that any change in the parameters or different asset the results go back to 50% win rate which is a random indicator baseline. I hope this helps.

@julianbonomini5883

11 ай бұрын

@@CodeTradingCafe thanks man! Will check it out! Love your videos btw

Hello. I’m really new in this world of programming, however I really want to start learning machine learning and coding in trading sphere, may you give some tips of the better way to start?

@CodeTradingCafe

Жыл бұрын

Hi 3 months python basics with exercises (see my codility playlist for few examples) then numpy pandas matplotlib for 1-2 months. At this point you can already build trading strategies without machine learning. When you feel ready start machine learning but this is a life long journey because there are new tools released almost every year. Good luck it gets fun with time.

hm based upon avaraged indicators predict..>> Something slow moving, stears something wildly moving.. this might not be ideal engineering for LSTM use. LSTM's need patterns, not noice. I'll analyse it deeper tomorow, as it helps me thinking about my tensorflow projects too.

@CodeTradingCafe

Жыл бұрын

Let us know how it goes, good luck!

Super material. Did you take into account to take additional data, i.e. I am working with BTC and to this I want to add data from the traditional market DXY and additionally sentiment with values for example from -10 to 10. How do you think it makes sense? I will treat DXY and sentiment as an additional indicator.

@CodeTradingCafe

10 ай бұрын

Thank you! I think your idea makes sense, I haven't included other prices in as indicators input, but this seems interesting to try. keep us posted, and good luck!

Thank you 👍

@CodeTradingCafe

10 ай бұрын

Welcome! and thank you for your support.

This is a valuable lesson about uncertainty and how much 'magic' one can squeeze from AI. I think you are asking too much from the model. Using the highest w/l strategies, not even the best traders can predict what the price will do at the *next* candle, for *every* candle. They are content with making predictions about the near, unspecified future, at times when many concepts converge on a bias. In essence, *nobody* can predict exact price movements at all times. Apart from making this demand, the worst 'crime' you ended up committing was throwing away volume information. Your model is tasked with price-action prediction, and you took away *its most critical* input. I don't know how one would train a model with the fuzziness of "tell me when prices will rise/fall, by approximately how much", which is what traders do. Probably best, however, would be to skip this approach entirely and task the machine to predict order signals, SL, and TP prices, then measure strategy performance, instead. And please, don't throw away volume.

@CodeTradingCafe

2 ай бұрын

I agree, it's too much to ask from an AI model or even a trader. I just don't believe in Volume when it comes to Forex (not centralized info), but yes volume is crucial when trading stocks and even Gold. I will get back to some strategies using volume and maybe backtest, had this on my mind for a while now. Thank you for your input.

Is it better to use LSTM on moving average curve? As there will be reduce noise in the data. Thanks a lot for your video!

@CodeTradingCafe

Жыл бұрын

Hi, yes and no, yes the noise is reduced but the moving average adds a lag to the related information, it might work better on high timeframes... but I don't think it's going to be any better (only a feeling opinion).

@musicxplore7926

Жыл бұрын

@@CodeTradingCafe thank you for your reply, that makes sense. With LSTM May I know what is the best accuracy you can get for the next day stock prediction? (Just in terms of trend, either up or down. I.e if random there should be 50% chance to go up or down)

@CodeTradingCafe

Жыл бұрын

This has better chances in my opinion, it might be a good video idea as well, ... Hourly timeframe and predict if the average future price is higher than the current price.

Great video. Do you teach somewhere online? Thanks.

@CodeTradingCafe

Жыл бұрын

Thanks, I don't teach online at the moment, I admit it's something that came to my mind, maybe providing full recorded courses, when time permits.

A question on lstm models. Do they care about data stationarity like regression models (eg ARIMA)?

@CodeTradingCafe

Жыл бұрын

As a first answer no lstm can deal with non stationary data, however I am now wondering if they would perform better on stationary data types... It's a never ending story with these stuff 🙂

@camstuart

Жыл бұрын

@@CodeTradingCafe haha! Yep! Forecasting from value difference as you have shown may be stationary naturally is what comes to mind 😊

Does that mean that predicting price fluctuations on a line that is not too attractive like that is not necessarily wrong? Because I was also studying and working on the same problem, and the results were similar!

@CodeTradingCafe

Ай бұрын

Well if it's not too attractive it might be right :)

Hi brother, can you please share with me a roadmap to becoming an algorithmic trader like you. I have already learnt python.

@CodeTradingCafe

Жыл бұрын

Python basics (including classes and inheritance), pandas and numpy, backtesting library (any, this is where you'll need classes). If you want to dive in machine learning it's a long way but for algo trading not worth it. And last but the most important... experience which can only be acquired through coding hours. Good luck!

Hi, you say its not working to predict the difference between the value. But what about predict whether positive/negative/zero Delta Predicting? Because I thinkt thats what matters to make money :D

@CodeTradingCafe

Жыл бұрын

Yes we are talking about a classifier which is a good idea to try, it might not work but it's good to see the results

Have you trained with a large amount of data, in the code it seems to be only a few thousand data?

@CodeTradingCafe

7 ай бұрын

The problem is where to get more data, and clean data without noise, it might simply be the bad application case for ML.

Какой объем видеопамяти лучше для lstm ? Вы на какой видеокарте выполняете вычисления ?

@CodeTradingCafe

Жыл бұрын

Hi I am really sorry Russian is one of the few languages I don't understand 🙂

@AliBaba__

Жыл бұрын

@@CodeTradingCafe Hello friend. I wanted to ask: what is the best amount of video memory for LSTM? And what graphics card do you use for LSTM machine learning?

@CodeTradingCafe

Жыл бұрын

@@AliBaba__ Hi, for this type of strategies like in the video we don't need a graphic card so far it's running on CPU power and it's enough, However if you intend to train larger datasets and more neural layers with high complexity models you can go for a simple graphic card 2GB memory should be enough. I hope this helps.

The problem in shifted values is in fact that regression model predicts value of Close very similar to close of current bar, not future ,even if you have targeted NextClose column with future close values. That’s why you have shifted values. So, they are very similar to previous closes … So I can’t understand why it works like this. Why people invented regression ML models if they work like this

@CodeTradingCafe

9 ай бұрын

Your description makes sense, but here we are only trying to predict one bar ahead, ideally we would try 5 or even 10 bars in the future, but this only works in different fields (climate forecasting) because there is a pattern followed by nature and the we do understand the pattern well enough to make some forecasting, even there it's also challenging. The trading market is so noisy these models are not able to see a specific pattern.

@vyacheslavfiodorov5738

9 ай бұрын

@@CodeTradingCafe Almost agree. However, you just need to figure it out. I tried various regression options by collecting last candle data from my broker. Knowing that the model can only predict the next value based on the last row (OHLC..), I also created the Features/Target column with a shift up one row. However, every time the values of the predictions of this column come out close to the current Close and not the next one. While the next prediction is close in value (~0.997 r2) to what should have been in the previous line. That is, the predictions of the random number have a close (~0.997 r2) value to what should follow next, but they are somehow shifted. And on your chart, on the video, you also demonstrated this. This breaks my brain. I can't understand why

"the one on the right is predicting the closing price of the next candle and the one on the left is trying to predict the price movement between the two candles" you have said. so meaning at the close of the candle lets say we are on 4h chart . if we use the model on the right to predict the market at the close of the 4 hour chart .literally basing on your explanation we are predicting the close of the next candle. so if we have a almost right predictions for the next close every time a 4h candle closes or daily candle closes using the model in the right cant we use that info to open trades

@CodeTradingCafe

9 ай бұрын

Definitely not! visually it looks appealing but it has no prediction power. To make it simple most of the times the algorithm is predicting the same close price as the previous candle and we would have this perfect look of predictions only with a slight translation. Again it looks so nice but in reality it's just copying previous closing prices. The problem is that blogs online show this to attract audience but it doesn't provide value.

I wanted to ask, do these work in real time? For example let's say i have a market structure script, it will work in real time?

@CodeTradingCafe

Жыл бұрын

Yes I did automated bots previously some are on this channel, although I don't recommend it, I prefer to receive phone alerts and I log in and execute trades myself instead of the bot, so the algorithm is only my assistant.

@Kay-qg1vn

Жыл бұрын

@@CodeTradingCafe okay i understand, if i should ask, why don't you recommend it? I was thinking of learning pinescript for the tradingview platform, have been considering it but haven't reach a conclusion yet, are you familiar with it? It is not a full fledged programming language but it get most of the job done, bad thing it doesn't allow external libraries or analyzing more than 30 symbols in one script.

@CodeTradingCafe

Жыл бұрын

@@Kay-qg1vn I don't recommend it because algorithms can be very powerful at some point and mess up at others (like existing a trade at the correct moment for example). I am not familiar with other tools only python, c++/metatrader.

You are certainly right. I have traded cryptocurrency in practice for 3 months. I do the swing trading and clearly realize that it is hardly to calculate or predict the oscillate range of any coin. But I think trend prediction would be a better direction. Transferring the problem from regression to classification may be more efficient🧐

@CodeTradingCafe

2 ай бұрын

Definitely much easier and more understandable, still challenging though once you start the code and the tuning.

thanks for this excellent video - so basically the LSTM RNNs just dont work for predicting future price action. Is that correct? I did try and rewatch why training it to predict pip differentials vs future spot price results in wildly different predictive value but dont understand your reasoning, Can you rephrase it for me please?

@CodeTradingCafe

4 ай бұрын

Hi, the way it's done in this video no it doesn't work. Basically the model is estimating the future by copying the current price which is its best guess.

@koipoi8111

4 ай бұрын

@@CodeTradingCafe As a proficient trader with significant knowledge of TA and fundamentals, may I make a suggestion that could increase the predictive accuracy substantially? Essentially there is a significant temporal issue here and the very premise of trying to predict next days close price based on previous 30 day data is flawed for a number of reasons (fundamentally, technically etc). Happy to also take this 'offline' and communicate via email if you prefer.

@CodeTradingCafe

4 ай бұрын

Sure! that would be great if you have time just either email or even here in the comments, let me know your thoughts now I am curious :)

@koipoi8111

4 ай бұрын

@@CodeTradingCafe sure what is a good email to reach you at? I will email you from my work email

@CodeTradingCafe

4 ай бұрын

Hi, codingntrading at gmail dot com it's in the about section of this channel, in case.

I have a question: to calculate EMAF or RSI you need some historical Data. How do you calculate the first row where there is no historical data? i don´t understand that

@CodeTradingCafe

Жыл бұрын

It's not calculated for the first few rows, if we choose RSI length 14 for example we start our first values from row 14, when all indicators are calculated we discard the first few rows from our dataframe to keep only clean rows. I hope this answers your question.

@mastermind2362

Жыл бұрын

@@CodeTradingCafe thanks for your fast respond and explanation! I understand that, but where exactly do the values in the first rows come from? Are they calculated oder just random values? Thanks again!

@mastermind2362

Жыл бұрын

@@CodeTradingCafe everything is clear now. I just commented the line "dropna..." out and know i get all this NaN values.

I'm having trouble getting data using yfinance on intervals less than 1d. It is impossible to get more than two months of any asset. If you have any ideas I would appreciate it. Very good video!

@CodeTradingCafe

Жыл бұрын

Yes it's limited to 60 days but you can make a loop to download many 60 days batches, it's annoying I know but they probably have a reason for limitations

@anderwork7571

Жыл бұрын

@@CodeTradingCafe can you make a video explaining how to correctly join the dataframes? Thanks a lot

@anderwork7571

Жыл бұрын

@@CodeTradingCafe If you have a csv collection of forex majors on low timeframes, I'd appreciate it if you'd share them with me. I can't get them at all. Thanks a lot

@CodeTradingCafe

Жыл бұрын

@@anderwork7571 I usually download most recent data using dukascopy, I don't have like a proper database, go for dukascopy it provides a lot of historical data you just need to create an account it's free.

@beastx4780

7 ай бұрын

@@CodeTradingCafecan you download sets of 5 min data for 60 days from y finance from over 60 days ago?

how to predict values for the next 10 days?

@CodeTradingCafe

Жыл бұрын

I could give you a beautiful shiny (but fake) answer, or I can tell you the honest truth :) predicting values 10 days ahead relying only on technical analysis is IMPOSSIBLE. But predicting a simple trend (which is more than enough for trading) is maybe possible only for the next day or 2!

What was the acuracy of prediction, in porcentage, aproximatelly?

@CodeTradingCafe

11 ай бұрын

Hi, depends on the limit we use for setting the target category, but in straight answer to your question around 52% if TP/SL = 1

How did you get the data?

@CodeTradingCafe

Ай бұрын

Yfinance, my broker, or Dukascopy...

how can we implement market structure such as wyckoff and elliot waves, how do we draw the lines?

@CodeTradingCafe

Жыл бұрын

Stay tuned 🙂

@Kay-qg1vn

Жыл бұрын

@@CodeTradingCafe please and thank you, i have been beating my head about this.

@CodeTradingCafe

Жыл бұрын

Might be a while though, to be honest I keep most of requests on a list but the list is faster than my video production...

@Kay-qg1vn

Жыл бұрын

@@CodeTradingCafe That's understandable honestly. The list is too long, nevertheless i am looking forward to it.

i think i have failed to understand you, but my question is if i run my model on the right at the closing of the trading day to predict the nextclose for the next trading day .cant i use that generated prediction to open trades since it predicts the market close for the next closing day. i mean if its on thursday 12:00 midnight or friday 12:15 am and i run my model on the right and i see that the candle on friday will close below the thursday candle ,can't i use those predictions to trade the market

@CodeTradingCafe

9 ай бұрын

Hi again, (check my answer to your other comment). In brief no because the predictions are really bad they only look good when we "zoom out" showing the full time slice of predictions, if we zoom in and check the details of the results the error is huge and not good for trading.

@muhireinnocent2371

9 ай бұрын

@@CodeTradingCafe thanks for your feedback actually i had created a trading bot that was using the model that's similar to the one in the right and it was making many losses , would adding more indicators make our model more accurate

@CodeTradingCafe

9 ай бұрын

Hi, no I don't think adding more indicators would solve the problem, it's more related to the model itself. Steer away from ML for trading, classic programming works better.

@muhireinnocent2371

9 ай бұрын

@@CodeTradingCafe when i use that code for currency pairs, in the target column i get zero for almost all rows as the change in price movement , what's causing that?

@CodeTradingCafe

9 ай бұрын

I am suspecting a scaling line somewhere, but need to go through the code to be confirmed.

Hello my age is 15 and have already started to master in python can any suggestions you can give me to polish myself?

@CodeTradingCafe

Жыл бұрын

I wish I could learn python at 15 but it wasn't there yet 🙂, first congrats you took a nice path, python basics, containers and comprehensions, pandas and numpy. Then if you want depending on how comfortable move to machine learning with sklearn starting with basic regression problems, linear and logistic. This is a plan for 2-3 years taking into account your math background at 15 years old. Good luck!

@hassanbutt8893

Жыл бұрын

Firstly Thank you brother there is no one to congratulate me and it means a lot to hear that from a credible person like you secondly I am learning from a book of python from basics to advance level oop,graphs,numpy,graphs and many other things in it and I have decided to learn from another book python and data analysis it is a bit advance but it will sharpen my skillset.My question was that can I learn it within 2 years and start trading?(Sorry for saying alot😐) Peace out

@CodeTradingCafe

Жыл бұрын

@@hassanbutt8893 yes you can in 2 years up to 3 years. It will definitely pave your way towards a data science specialty

@hassanbutt8893

Жыл бұрын

Thanks bruda👍

1- If you calculate power of predicted results few times looks like you will get barely enough results for price change. 2- Instead of predicting how many pips price gonna change, predict how much percent price gonna change rational to the last candle. In other words don't confuse our little brained neural net with exact numbers or pips, instead teach it to predict relatively change of price as daily percent.

@CodeTradingCafe

7 ай бұрын

Hey, thank you. Your proposition makes perfect sense, I did something similar in another video it's still challenging but needs more time to make it work.

Why dosnt it soft the sa though I've watched it over and over so many tis?

@CodeTradingCafe

Жыл бұрын

I am not sure I understand your question sa tis?

do this and make a series pls: # Data Acquisition function gather_data(): collect_stock_data() collect_news_data() collect_social_media_data() save_data() # Preprocessing function preprocess_data(): clean_stock_data() clean_news_data() clean_social_media_data() process_text_data() # Tokenization, stemming, etc. save_preprocessed_data() # Feature Extraction function extract_features(): compute_technical_indicators() compute_sentiment_scores() create_feature_matrix() save_feature_matrix() # Time Series Analysis function train_time_series_model(): fit_ARIMA_or_LSTM_model() save_trained_model() # Ensemble Modeling function train_ensemble_model(): fit_GBM_model() fit_SVM_model() fit_Random_Forest_model() combine_models() # Bagging, boosting, or stacking save_ensemble_model() # Reinforcement Learning function train_trading_strategy(): implement_Q_Learning_or_PPO() optimize_trading_strategy() save_trading_strategy() # Model Evaluation and Selection function evaluate_models(): compute_MAE_or_MSE() compute_Sharpe_Ratio() compare_model_performance() update_ensemble_model() update_trading_strategy() # Continuous Learning function continuous_learning(): while True: gather_data() preprocess_data() extract_features() train_time_series_model() train_ensemble_model() train_trading_strategy() evaluate_models() # Monitoring and Alerts function monitor_performance(): track_model_performance() track_trading_strategy_performance() send_alerts() # Main Execution function main(): gather_data() preprocess_data() extract_features() train_time_series_model() train_ensemble_model() train_trading_strategy() evaluate_models() continuous_learning() monitor_performance() main()

@CodeTradingCafe

Жыл бұрын

Hi thank you for taking the time to present all the details, this is a lot of work though and I am not sure a video with all of this is something most people can follow, I might however take some parts of your proposition. Thanks again I'll see you around :)

@domenicperito4635

Жыл бұрын

@@CodeTradingCafe cool

Good

@CodeTradingCafe

3 ай бұрын

Thanks

@abbaali5900

3 ай бұрын

Sir I need the source file please

@CodeTradingCafe

3 ай бұрын

Hi check the description of this video kzread.info/dash/bejne/mqSas6dpkrfLcZc.html

Just compare all model avec naive forecasting : price tomorrow = price yesterday and the results will be more interesting then any other model

@CodeTradingCafe

Жыл бұрын

Interesting idea! didn't occur to me to compare with previous day price.

When are you going make next video while improving on this ??

@CodeTradingCafe

3 ай бұрын

Left it for a while and the list is getting longer, might not be anytime soon unless I discover an interesting idea I will share it.

What kind of loss function are you using? Remember one of the most common loss functions, MAE, will yield the same error regardless of whether your prediction is above or below the true value. For example: todays closing value is 6.0 and tomorrow the closing value will increase to 7.0. If using the MAE loss function, then, say a prediction of either 5.0 or 9.0 will yield the same error (they are equally far away from 7.0). MAE doesn't care that we want to know the direction of the movement, and is equally happy giving us the answer of either 5.0 or 9.0. Although they are the same in terms of absolute error, one if them is a prediction in the true direction, and the other is a prediction in the false direction. But in a real life scenario predicting the direction of movement (up or down) is just as (or even more) important than arriving at a low error. Well, that is my two cents at least. You have to get creative constructing a custom, asymmetric loss function that punishes the model not just for the absolute error but also for predicting in a faulty direction of movement. If you figure this out, please make a video about it and I will perhaps "borrow" your loss function! xD

@CodeTradingCafe

Жыл бұрын

I am not ignoring your comment I just didn't have a moment to review the code... will get back to you from the mess of the endless files on my desktop 🙂

@CodeTradingCafe

Жыл бұрын

... using the mse, I see your point no turn around at this point, unless a simple difference function. And to be honest I don't think it changes much.

when are you going to make next video to this series

@CodeTradingCafe

3 ай бұрын

Do you mean revisiting neural networks?

@maciejkolibabski7491

3 ай бұрын

@@CodeTradingCafe evaluating the model in order to obtain the better predictions results

@gamingwithvillain1731

3 ай бұрын

​@@CodeTradingCafe where are you going to make video on improving on above mistakes?

It is impossible to predict a value, but predicting a probability of up and down move has a chance.

@CodeTradingCafe

Жыл бұрын

True, it's easier to guess a trend, we will see how to sort it out... But the lstm was appealing to try, neural networks sound fancy 🙂

@eto38581

Жыл бұрын

I spent 4 months trying to predict if bitcoin would rise or fall based on previous candles. I used many indicators and dynamic range of backcandle values. The result was %48 success at best. So flipping a coin has better chance.

@JonCianci12

11 ай бұрын

Just reverse the signal😂

@eto38581

11 ай бұрын

​@@JonCianci12 Hahaha, so if I make a %5 successful model, I can just reverse the signal and win %95 of the time 😀😀

Hi, I just watch this video, is there a new update about that?

@CodeTradingCafe

Жыл бұрын

Hi, thank you for your comment, I haven't done neural networks lately, I was thinking reinforcement learning deserves a try... maybe.

@luchogofre

Жыл бұрын

@@CodeTradingCafe great video! You teach excellent

i tested this way, it does not work.you can divide close price to open price, then use it for target and also features. but it does not work too.

@CodeTradingCafe

Жыл бұрын

Lstm has a lot of struggles with this type of random data

but we just need the trends not the actual price right. maybe the profit or loss would be less or more but we can surely get if it would be a profit or loss and trade only when profit is predicted

@CodeTradingCafe

Жыл бұрын

Well if we can't get the price I will go for the trend, so we need to modify the code for this maybe I will think about something related.

@rickmondal5126

Жыл бұрын

@@CodeTradingCafe maybe i can help

@CodeTradingCafe

Жыл бұрын

I was thinking just predicting the future average price for the coming let's say 5 candles... Only predict if it's above or below current candles close so basically it's a binary classification it should be easier this way.

why to limit LSTM. It can use any size of memory. I think no need to set any number of candles to limit... If you use that limit then just use usual ANN instead of LSTM

@CodeTradingCafe

2 ай бұрын

Definitely need to limit the backcandles even for LSTM otherwise we would be fitting on the whole set of data meaning also data from very long in the past which will not influence current behavior in reality, so past data becomes noise and hinders the model.

Sorry but i don't understand why first pred are wrong, someone could explain me?

@CodeTradingCafe

Жыл бұрын

Because the model is choosing the same values as previous candles close with a small shift.

@williamlacerra1835

Жыл бұрын

@@CodeTradingCafe thank you for reply, still confused, so if i test It with real time data, to predict next 10 candles for example i still get mistakes i guess.

@CodeTradingCafe

Жыл бұрын

@@williamlacerra1835 yes definitely, this was a demo showing how tricky machine learning can be

@williamlacerra1835

Жыл бұрын

Please continue with this series very interesting and useful!

@CodeTradingCafe

Жыл бұрын

@@williamlacerra1835 more is coming but it takes time to make videos with concise content

Tried lstms on market things and it was terible. I dont think there is much to do on it from just price data.

@CodeTradingCafe

Жыл бұрын

Yes my observation as well it doesn't work for this type of data, too much random noise, the model is lost.

@nevokrien95

Жыл бұрын

@@CodeTradingCafe I think the only real sucess i have heard of in this area is using deep languge models on news. The fact u can use a pretrained one REALLY helps with the small data problem

@CodeTradingCafe

Жыл бұрын

Ah yes language processing might do the trick

@nevokrien95

Жыл бұрын

@@CodeTradingCafe honestly anns have not showen good results on time seiries data compared to arima. I have tried it myself and i gota say the results r awful

mb after reconstruction ts lose patterns

@CodeTradingCafe

11 ай бұрын

So far neural networks are not performing well on market patterns, simply because these are not periodical and have a lot of randomness.

1st of all you need much more candles to train such model maybe a few or several thousands, 2nd features are not correct. You predict return, so features should be e.g. lags of prices not absolute values of EMAS which are not stationary so perhaps lags of emas plus lags of prices plus more candles will help. To compare just calculate RMSE with/without those changes to see if there is any improvement.

@CodeTradingCafe

Жыл бұрын

Hi thank you for your input, I think we have couple thousands of candles I believe it should be fine since it's not the priority looking at the results. For the lags of prices you might be right I could try this with lags of emas as well and see what it gives but I don't expect any different results, lstms are very tricky to handle.

It’s all pure random walk It’s the reason way hedge funds use arbitrages and hedging instead of spending time to beat the market The idea is to see some errors or some extra situations that give certainty that the price level will join a normal values That how professionnel traders works No one of them try to predict the future, it’s just impossible

@CodeTradingCafe

Жыл бұрын

Deep down we know it's true but curiosity drives us differently sometimes. However you can always join an ongoing trend for a short time.

First, input data must be *stationary* Second rule, it must be *stationay*

@CodeTradingCafe

Жыл бұрын

Hi, input data is stationary here, e.g., RSI values are always between 0-100, the price differences as well between 2 limits, we didn't use absolute values for any of the features, even with these considerations it didn't work, the randomness of the market can't be adjusted into a model I think the whole approach here is not optimal.

@sgrouge

Жыл бұрын

@@CodeTradingCafe I used to play with keras on EURUSD at tick lvl. I have attained 73% precision with a wavelet compression, for the next 1 minute movement. So at high resolution, market is predictible. I think guessing the next move is a wrong problem since most candles are random, and some others are predictible. the core problem is filtering the noise, aka trying to predict the predictible move, and throwing the non-predictible one. For instance there are moves according to liquidity pockets on the market. But keep on the good work bro.

@CodeTradingCafe

Жыл бұрын

@@sgrouge 73% is quite impressive using NN. I like your idea on filtering only what is predictable, makes things better.

@TungPham-di9yk

9 ай бұрын

@@sgrouge Can you explain me a bit about noise filtering? I know that there are some predictable patterns that would give a good signal, however there are just too many indicators out there that I'm just lost while searching for the effective ones

Professional bank trader here. You can make money on market ONLY by eliminating market inefficiencies, not by predicting prices, it's impossible. There is no way to find those only with the chart data. You need full order flow in real time. If people can trade with positive expected value, than algorithms will too. But you have to be a trader yourself, to build a working one. Imagine being a profitable trader and a machine learning engineer at the same time, a unicorn.

@CodeTradingCafe

7 ай бұрын

Hi thank you for your input, I agree if a trader does it than a an algorithm should as well in theory, the only issue put all your skills as a trader into an algorithm... it's possible only it takes some time. The unicorn... you can craft it in 2 years time with a good teacher :)

I presume this whole quant thing is just a waste of time. Looking at results i guess i am not going to make money with data science skills in this life.

@CodeTradingCafe

2 ай бұрын

No, don't get me wrong, the neural networks thing doesn't work for trading (out of experience and discussions with quants), but algorithmic trading there are plenty of ways to assist in making money, it works, just don't imagine it can turn 1000$ into 1000000$ in couple of months, it's not gambling it doesn't work this way and sometimes it's really slow.

the problem is overfitting

@CodeTradingCafe

Жыл бұрын

One of many 😂

A why i never see in those type of videos people using something other than Japan candals. I mean, come on! Why aren't you using data from order book or volume or eitherscan or some other information that have a significant impact on the price? The price chart is a SCAM. Those fancy technology does absolutely nothing to improve our chances. It's still no better than flip of the coin.

@CodeTradingCafe

2 ай бұрын

Yes and no... in forex volume is meaningless because it's broker related, same for market depth level 2 data, these only work well for stocks and futures. The price chart is not a scam it's smoothed data for me, the raw tick data is too noisy, just smoothing it can make things easier, and works really well for high timeframes for example 4H and daily these are really easy to trade, albeit slow so you have to be patient and just wait for opportunities, this is where algorithms come into play.

Please, god transformers better lstm bad

@CodeTradingCafe

Жыл бұрын

It might be worth trying transformers but maybe they are data hungry.

@axelanderson2030

Жыл бұрын

@@CodeTradingCafe feature engineering works well for that, add financial indicators to the features.

Even your tip is bad :) on the right you got lag Pointless data. Your prediction must be before real data or on the same index.

@CodeTradingCafe

Жыл бұрын

Now I am curious which tip from the video :) but yes a look ahead bias is the main pitfall

Both models are crap...

@CodeTradingCafe

5 ай бұрын

Hi, I don't which models exactly, this video doesn't present any specific models?

Finally someone said it. The LSTM prediction is as useless as doing df['pred'] = df.close.shift(1). It thinks tomorrow the close price would be the same as today.

@CodeTradingCafe

Жыл бұрын

Thank you for your comment, I found better results with normal ML classifiers (still not enough for a descent trading model), and even better results with simple NON-ML models using classic price action detection! ironically :)

@andreibudaes3966

Жыл бұрын

@@CodeTradingCafe yes same here. I tried a bunch of stuff including using the openai api (gpt-3.5 model) but the backtest shows I actually would have lost about 40% of the initial sum...keep trying...there's bound to be a good ML strategy...I was actually thinking to get historical financial news data, get a sentiment analysis and clump that in the LSTM somehow...I've also looked up the gym-trading lib and reinforcement learning. Here's something I've just watched about it. kzread.info/dash/bejne/dm2nt5OhfLeWgbw.html. If you get some good results it would be amazing. I'll let you know if I get any good results too

@CodeTradingCafe

Жыл бұрын

Thank you for sharing this, I haven't seen this video! will take a look at it, just for the fun of RL.

@andreibudaes3966

Жыл бұрын

@@CodeTradingCafe no problem. another thing I was trying was using fbprophet (now just prophet) to get weekly and daily seasonality for the price movement...found some interesting results but not so much for signal classification but rather for deciding how much to invest as a % if a signal does appear.

@CodeTradingCafe

Жыл бұрын

@@andreibudaes3966 that's interesting, someone else did mention fbprophet in here I have to add it on my todo list.

how to get real prices in this code, how to use (sc.inverse_transform())

@CodeTradingCafe

Жыл бұрын

the inverse_trasform takes the scaled values as argument and returns the unscaled real prices.

@onlinerender9084

Жыл бұрын

@@CodeTradingCafe Sorry how to implement this code: y_pred = model.predict(X_test) #y_pred=np.where(y_pred > 0.43, 1,0) for i in range(10): print(y_pred[i], y_test[i])