TCC I - Wesley Barbosa Silva

[00:00:00]Hello, [music] [music] [music] [music] everyone. Good morning to everyone who is here following the defense of student Weser Barbosa da Silva's TCC1 (Final Course Project 1). This defense, this work from ES, was supervised by me and Professor Caminho.

[00:00:47]Professor Caminho is a professor at FEAC, right? The Faculty of Economics, Administration, Current Affairs and Accounting at UFC. The panel consists of Professor Bruno Richelli from the Federal University of Ceará, Crateú Campus, and Professor Rafael Braz from the Department of Statistics and Applied Mathematics, also from the Federal University of Ceará. Well, it had fallen, but it came back, right? See you later.

[00:01:13]The internet on campus.

[00:01:15]Ready. Please share your presentation again.

[00:01:25]Okay, now you can see it perfectly. It's charging perfectly.

[00:01:30]Eh, the work hurts, folks, the title is modular library for forecasting temporal series with large models, with large-scale language models.

[00:01:40]Uh, this is Wesley's TCC1 project.

[00:01:43]Wesley, according to the TCC1 regulations, you have up to 20 minutes to present your work, right? Yes, and you can start right now, right? Hey, good morning everyone. My name is Wesley Barbosa Silva and I will be presenting my final course project, entitled "Modular library for forecasting in time series with large-scale language change". This work is supervised by Professor Dr. José Wellon Franco da Silva and co-supervised by Professor Dr. Carlos de Oliveira Caminha Neto.

[00:02:15]This presentation is divided into five parts: introduction, theoretical framework, related works, methodology, and references.

[00:02:25]So let's go. Well, time series forecasting has applications in various fields, such as finance, retail, urban mobility, and healthcare. This image, for example, from the Ministry of Health shows the evolution of COVID-19 cases in Brazil between March 16 and 20, 2020. And it exemplifies what a time series represents and, even more so, the importance of time series forecasting, since during the pandemic, estimating the number of new cases of the disease was very important for planning hospital beds, medications, and medical teams, for example. So this ability to estimate future values ​​based on historical data, well, that's exactly what helps in decision-making, reducing uncertainty, and optimizing resources.

[00:03:16]Traditionally, this field has used statistical models such as ARIMA, machine learning models such as Candle Forest Gusto, and also deep learning approaches such as LSTM.

[00:03:30]However, in recent years, large-scale language models, or LLMs, have demonstrated a strong ability to generalize to non-textual data. So, some recent studies already employ LLMs in tasks related to time series, such as anomaly detection, classification, time series generation, among other types of related tasks. And more recently, these models have opened promising avenues in time series forecasting, such as predictive models, offering more flexible alternatives when compared to traditional architectures.

[00:04:08]And although there are robust libraries that are widely adopted in the literature, they tend to prioritize only classical statistical models or machine learning approaches.

[00:04:19]Currently, no library offers support for general-purpose LLMs, such as GPT and Gema, for predicting temporal series as predictive models. Well, why develop a library, anyway? The main focus of the library is researchers who, by using the library, can perform comparative analyses with traditional approaches through Marx's work. And in addition to researchers, developers who use the library can also develop interactive applications such as web applications and mobile applications, for example.

[00:04:54]Therefore, given this context, the overall objective of this work is to develop a library to integrate LLMs (Low-Level Meteorological Systems) into the forecasting of temporal events. with support for PROMP engineering and comparison with traditional methods. There are three specific objectives for this work. The first is to develop a modular library in the Python programming language for integrating LLMs into time series forecasting. The study also aims to investigate the impact of different engineering techniques on LLMs and, finally, to compare the performance of LLMs with approaches established in the literature.

[00:05:28]Before defining what a time series is, I'll give an example of area two emissions worldwide. So this table on the right shows two pieces of information. The first is the date on which an observation was recorded, and the other is the amount of CO2 in millions of tons on that date. So, for example, on January 1st, 2026, 103.7 million tons of CO2 were recorded. The following day, 109 million tons of CO2, and so on, day after day. So, a time series is nothing more than a sequence of observations measured over time at regular intervals, uh, of regular times. And these time intervals can be annual, quarterly, daily, hourly, etc. So, if we revisit that example of CO2 emissions from areas, we have a daily time series.

[00:06:23]Furthermore, it is also important to emphasize that, unlike regular data, the order of observations matters.

[00:06:30]So here I like to draw an analogy with the film. So the movie's credits scene has to be at the very end of the movie. So if for some reason the credits scene is in the middle of the movie, it wouldn't make any sense at all. Therefore, in the same temporal way, it needs to respect the chronological order of the observations.

[00:06:48]And how can we represent a time series graphically, right? Well, if we take that example of emissions from salt flats around the world, on the X-axis we have the time axis and on the Y-axis we have the axis of our variable of interest. One very important definition is that when we have only one variable measured over time, this time series is called a univariate time series, but we can also have two or more variables measured over time. Therefore, in this case, the time series is called multivariate.

[00:07:21]So, in this example, we have CO2 emissions and energy generation based on heat.

[00:07:27]And what is a weather forecast? It is simply the process of estimating future values ​​based on past observations. So, based on historical data, a model will make estimates of future values.

[00:07:41]So, for example, the model can predict a week, it can predict a year, or it can predict a month. And this number of future steps that the model will predict is what we call the forecast horizon. And this work employs both statistical models and artificial intelligence models. That's why it 's very important for us to define what artificial intelligence is, which is nothing more than the field that develops systems capable of performing tasks that require human intelligence, such as pattern recognition and decision-making. Within artificial intelligence, we have several sub-areas such as machine learning and deep learning, which is where large-scale language models are found, and that is the focus of this work.

[00:08:26]But before we talk about large-scale language models, it's very important that we define what machine learning is. First, it is the subfield of artificial intelligence that develops the methods and algorithms that allow systems to learn patterns from data without the need for explicit programming. So, if we look at these two images, we have a comparison between the traditional approach and the machine learning-based approach. So, while in the traditional approach the expert needs to write the rules explicitly, that is, they need to make a list of instructions that the algorithm must follow rigorously, in the machine learning-based approach, the expert will train an algorithm with the data and from that the algorithm will solve the task automatically.

[00:09:17]Within machine learning, we have several learning paradigms, among which the supervised learning paradigm is the one most directly related to time series forecasting. This paradigm simply involves training the algorithm with labeled examples and enabling it to associate each input with an expected output. Within this paradigm, we have two types of problems: a classification problem and a regression problem. So, in the classification problem, we'll have labeled instances, and the goal is to predict the label for a new instance. So, imagine, for example, that in your inbox you have both desired emails and unwanted emails.

[00:10:00]Unwanted emails are those emails, for example, marketing emails or scam emails, and from there you will label each of these emails and train an algorithm with them. So, from that point on, when you receive a new email in your inbox, the algorithm will be able to determine whether that email is wanted or unwanted.

[00:10:23]In regression problems, the goal is to estimate a continuous numerical value based on past numerical values. Therefore, we can formulate the time series forecasting problem as a supervised regression problem.

[00:10:41]Now, finally talking about large-scale language models, the LLMs, right? What are they? They are models that are trained on a large scale, right, on large amounts of data, and they exhibit emerging capabilities of comprehension, reasoning, and text generation. So how does an LLM work, right? Well, starting from an input that we call a prompt, for example, what is the capital of Brazil? From this point, the first process that occurs is called tokenization, in which this input will be segmented into several blocks. And we call these blocks tokens. Each token, in turn, will have a numerical identifier. And each numeric identifier will be represented by a vector representation, which we call an embedding vector.

[00:11:31]These embedding vectors will be the input to the model, and from this, the model will calculate the probabilities of the next word.

[00:11:40]And from the moment the model determines which word is most likely, it will pick that word.

[00:11:47]So, for example, when asked what the capital of Brazil is, the most likely answer is Brasília. So this word, this output of the model, will be appended to the input, and this entire process, this entire flow, will repeat itself again, which is called an autoregressive process, until the model reaches a stopping criterion that is usually somewhat special, indicating the end of the sequence.

[00:12:12]Therefore, this ability of the model to perform in-context learning reinforces the importance of the role of program engineering, which is nothing more than the process of building and optimizing instructions to maximize the effectiveness of artificial intelligence models' responses to specific tasks. So, for example, instead of asking what the capital of Brazil is, you could ask a more complex, more elaborate question. And for that, we have some strategies that are called "ready-made strategies." And in this work, he adopts three main strategies from the ready. The zero-shot strategy will define the role of the model. So I say that the model is an expert in time series forecasting. Furthermore, it will define the model's objective, which is to predict future values ​​based on historical data. It will also provide a statistical context for the model, as well as the steps the model must take to perform the task, and finally the historical data itself. The F-shot strategy complements the zero-shot strategy, but we 'll add some entry and exit examples. The input is the historical values ​​and the output is what a good forecast would be. So, we're going to give the prompt several examples, so that the model can better understand these patterns. And another strategy, COT, which is short for chain of thought, consists of adding intermediate reasoning instructions, causing the model, before generating the forecast, to analyze some important aspects of the time series, such as trend, seasonality, outliers, and cycles.

[00:14:10]And in addition to prompt strategies, it's also very important that we talk about formatting.

[00:14:17]So, this 2024 paper made a very important discovery, demonstrating that data formatting has a significant impact on model performance. So, when you have poorly formatted data, it tends to hurt the model's performance.

[00:14:34]So, in this image, we have the performance of GPT5 for three data representation formats. And we can see that the CSV format performed better when compared to the JON format.

[00:14:51]Well, in addition to the Promise and formatting strategies, tokenization is also very important. So, if we take, for example, the GPT5 again and the model, for example, Pen 3, and we pass our data in numerical representation, that is, in the conventional representation with which we pass the data, we will realize that the GPT5 and the Pen3 will tokenize the data differently. And that's to be expected, considering that each model has its own tokenizer. So, for example, the number 151, GPT5, it will create a unique token for that number, while TR will create a token for each of these. So, there will be three tokens for the number 151. And what does this article bring to us that's interesting, right?

[00:15:43]He brought an approach that consists of adding some spaces between each of these numbers, causing us to force the model's tokenizer to create a token for each G. So, for example, GPT5 in this textual representation will have a tokenization similar to P3. And we can see that this led to an improvement in his performance. This article was published in Neurips in 2023 and has 1084 citations.

[00:16:16]Of the works listed here, we have the five main libraries of the temporal census. And this table compares these libraries in relation to support for statistical learning models, deep learning models, foundational time series models, and LLMs. So, of those five libraries, it's worth highlighting SK Time, which has partial support for LLMs, considering it only has a very specific type of LLM, and also highlighting Time ProPilot, which has support for LLMs. Yes, however, there is a big difference between Time Copilot and this work, since Time Copilot will use LLM as an agent and our work will use LLM as a predictive model. So what's the difference in Time Copilot? The time series analysis will select either a statistical or machine learning model to make the forecast. While this work will utilize the LLM itself to make the prediction. So, that's why we stand out.

[00:17:26]And the proposed architecture, well, it's divided into six modules. So, in the data module, we will support both univariate and multivariate time series. In addition, statistical calculations and some data preprocessing operations, such as imputing missing values, for example. And in the models module, we will support both BAP models on the Cloud and local models, running through LM Stud, for example. And in the prompts module, we're going to support the main prompt strategies: no zero shot, free shot, call of duty, as well as combinations of the two, and also custom prompts. And in the metrics module, we'll provide support for the main established metrics from the literature, such as My, RMSE, and USMP.

[00:18:16]And the formatting module will support structured, textual, and custom dates. And in the formatting module, the main types of charts, right, like line charts, bar charts, as well as tables too.

[00:18:31]And from the selected dataset, there are three. Well, the first one is ETTH2, which is a very classic dataset, used by many weather forecasting articles, and it consists of oil temperature forecasting. It is a time series. And the other is Electricite, which consists of forecasting a house's energy consumption and is a daily time series. And finally, the carbon monitor, it is also a dataset, a time series also by area, whose objective is to predict CO2 emissions from different areas around the world.

[00:19:11]Of the models that will be used, they are divided into categories.

[00:19:15]So, we have the naive methods, we have the classic statistical models like Altarima and Autts, and also more recent statistical models, like TETA, which won the M3 competition, and Profet, which is the model developed by the company MET. Yes, we'll also have machine learning models like Handle Forest and Shbost. In machine learning, we have LSTM and DCN. And from the LLMs open rate we're going to have the 332B. and the GMA 3 27B is picking up LLMs of different sizes.

[00:19:51]So, these are the promos that will be used, right? Zero shot, F shot, and chain of TCH.

[00:19:58]Well, among the formats that will be evaluated are the PL format and the CSV format, and we will also explore numerical and textual tokenization strategies.

[00:20:10]This is my schedule of activities. So, 2026.1. I completed these first six stages of the schedule, and in 2026.2 I will conclude the execution of the experiments and analyze the results. Here are the references, and that 's it. Thank you very much for your attention.

[00:20:31]Thanks. Thank you, Weser, for your presentation.

[00:20:36]He finished exactly on time, 20 minutes. Okay, that's pretty sharp. Congratulations. Okay, everyone, now we're going to start the oral examination part, right, with the panel. Hey Wes, could you go back to the first slide, please?

[00:20:53]Ready. Just so we can... Ready.

[00:20:59]First, first, first.

[00:21:02]Ready. That's it. From there, the teachers will ask you to go forward, backward, and clarify any doubts. Okay everyone, let's begin the questioning portion. Thanks again to the committee, Professor Rafael, Professor Carlos Caminha, and Professor Bruno for accepting the invitation. So I wanted to know what it's like behind the scenes, if we start with the most external professor, right?

[00:21:24]So I wanted to thank Professor Rafael for accepting the invitation to be on his panel.

[00:21:31]Oops, sorry, the internet in the field is really bad.

[00:21:35]Okay, but are you listening to me now?

[00:21:37]I'm listening now.

[00:21:39]Ready. So we're going to start with the most external meme, right, which is Professor Rafael Braz, who is a professor at DEMA, the Department of Statistics and Applied Mathematics here at the Federal University of Ceará, and who will be the external evaluator of your work.

[00:21:52]Professor Rafael, thank you very much for accepting the invitation, and feel free to make any comments on Wesle's work.

[00:22:02]Hey, good morning, everyone. Good morning, Wellington.

[00:22:07]Wesley and Bruno Carlos.

[00:22:10]I am truly happy to have been invited to participate in this panel. It was good, I only went once, right? I'll try going more often now too.

[00:22:22]So it's truly a pleasure to be here. First point, and I think this is the most important point, is that the work presented is of very good quality.

[00:22:35]Just to start, to put your mind at ease with any kind of thought. It's great, I liked it a lot.

[00:22:41]The writing is very good, the whole thought process, the whole way of thinking as well.

[00:22:49]So, with a few points that I'm going to share about the work, let's say, in order to contribute, I'll also send you some material that I've already annotated, that I did on my tablet, so I'll send it today.

[00:23:04]And the work is on a level, let's say, that will be very useful for future work by students who are interested in continuing in this field. No. So he's in the part that has a lot of information and self-help. That's very good, especially with the monograph. So it's at a very good level. Of course, there are certainly still things here that are close to what could be done in graduate school. I think that in the end there will be work that will certainly generate other publications, whether in conferences, events or articles. Of course there isn't one, right? As you put it, yes, it's more common to participate in events. So, the first point I wanted to make, along with a few more points that I think define—uh, it would be an intersection—a clearer definition when you put it there, right, where you put it.

[00:23:59]There are some points that you'll go into, let's say, in great detail, and other points that perhaps deserve special attention, right?

[00:24:06]For example, I defined what white noise is, right? You put the white river there and don't define it very well, you don't define what the white river would be. Of course, it's already very clear regarding temporal matters because it's within that field, but I believe your material is in a format that makes it useful even for those unfamiliar with the area, right? So, Rio Branco's definition, there's a point here when you introduce the progressive models.

[00:24:36]It's also good to include this constant here, because this little constant here will also be estimated, right? Because it seems that the estimate is based solely on the form used to set the autoregressive coefficients.

[00:24:48]So there are a few points being raised here that are important, even for the future. For example, when you mention the Arima model and its particularities, you make it clear in the text that it's there, it's a feature of Arima, it's there for something very specific to integrated auto models, right? It is a specific type of non-seasonality.

[00:25:16]So, make it clear in the text that it only covers something very specific, which are models that can become stationary through differentiation, which sometimes came across as if the author, on a page here, states that for non-stationary series, the weapon is used, but it's not for very specific series of non-stationary models. So, I just left this here for the contribution part, right, regarding the formatting, just like the part you put here about differentiation, right, on the wall is the model, the integration. So, there are some conceptual things that enrich the work a bit more, but everything will be laid out for you here, right?

[00:26:04]The part about "until when" you put there is also very cool from the substonencial perspective. There's a mathematical aspect to this that involves specifying the sum completely. You added the three dots. I think that since you know his formulation, you can put the correct mathematical formula.

[00:26:22]So, there are these points here that I'll start with; they're simple things, and they're more in the sense of making it closer to the correct reading, without a doubt, right? This book can be read by anyone in the fields of statistics, computer science, economics—anyone interested in the area can read it very well. Okay, let me just put the material here a little bit in front.

[00:26:56]One thing you mentioned, even in that graph you included—I think you can even look at it here—are the CSV, Jon, and TXT files. I think I understand what you're trying to say about better or worse adjustment, but you don't comment on it. Why, for example, is 'a' in that graph, meaning that Jon, for example, didn't have a good fit? That reddish outline there is like the prediction he made. What it means?

[00:27:32]Because when I look at that graph, I know the format is better. Just by glancing at the graph. Can you put it there?

[00:27:41]Can you tell me that?

[00:27:45]And what was the question regarding the figure that has the series and the input formats? Jon, TXT, and CSV. I know.

[00:27:59]It's about how you look at it, because you showed the graph, you already said there's one before this one, and this one too. But that one will do too.

[00:28:08]So, you just put that CSV is better or better, or Jon is worse, but that's all you put. You didn't explain what happens to make him better. So what happens? It's good because even though you can infer it from the graph, I think it's important to include it in the text, but what exactly was in that [ context]. Well, regarding why the model outperformed CSV, it's something we only do empirically, because each model will have a different impact for each format. There is no single format that is universally best for all models. That's what this article brings. Well, but the CSV format, what I think, what I believe, is that the data is predominantly in CSV format on the internet, when we go to access these datasets. So, I believe that LLM, due to being trained on large volumes of data, is mostly trained on in CSV format. So, yes, that's something I believe. So, I think for him, CSB is a better format, not necessarily. There is consensus, and when you glance at this figure, or what it means to be better, because what seems to be happening here isn't even about saying why or what generated this, it's about analyzing the result.

[00:29:43]It seems that here, for example, with Jon, we have a reddish hue here.

[00:29:50]This coloration, this red coloration, signifies greater variability; these are intervals that lead to something greater.

[00:30:00]Ah, so I think that's it, but it 's good to point out, I'm not trying to say what led to this, I'm trying to say, uh, that you glance at it and you come to the conclusion that the article reached. He had, you know, in that aspect, which I think is important to put in the text.

[00:30:20]I understood. So, uh, it's clear now, you understand, Weser? Well, because the figure doesn't... what does this figure represent, right? This redder one, this more expensive red. It's good to say, look, Jason's shape, it seems redder, it seems worse than the other shapes. What does it mean to say that? Do you understand? Mobility, in the end, did not converge. Well, it's about trying to make that clear in the explanation or even in the image, right?

[00:30:47]That's exactly it. I've already read the figure here and I can tell you why, because I think if this red color is here, then it seems that the estimation, the prediction, wasn't better, or since it's not just one prediction, it seems that the prediction had more variability, that it was more wrong, right? And it's the same thing with the next figure, right? That should be the same thing in this idea. So, I think it's good for you to understand this and write about it.

[00:31:20]Right? So for this part, I'll continue here, but everything's alright, right?

[00:31:28]You understand, right? What did I want to include?

[00:31:30]Yes, I understood. There's one here, let me see, I wrote it down just to bring it to you.

[00:31:44]There's very little written down because it's really well written, it's really, really good.

[00:31:52]Now, I'd also like to raise a point regarding the choice of dataset. I think it needs to be a little clearer, for example, when you input these datasets, because there's even the electricity part, right, where you chose customer 293, which I think is more of a choice.

[00:32:18]You can choose what helps you best, but you have to tell me what led you to choose that, whether it was done randomly or because this one has already been studied by other places, because that client, and also because these series are important, for example, what kind of series are you choosing? These are stationary series; it is an integrated series; it is a series whose trend structure or seasonality structure is not well identified by usual statistical models.

[00:32:53]It's good for you to get to know it a little from another source to understand why you're using it, because normally, you have that series where you believe that the model already has a very well- made fit from some other model, right? For example, the ARIMA model, the Sarima model, which is already widely used, has another series where the Sarima models, or the exponential aviation model, don't fit well, and your model might be more flexible to suit that series.

[00:33:25]So these series here were probably used in other works in this line of research, but even if you look for a series from another work that shows this distance from the usual models, it will show that the model, your model, captured that structure that was not, let's say, expected by these models that you put here. So, in that respect, how did you come across these series?

[00:33:58]Well, it was mostly based on work I'd seen, so I selected it specifically to find work related to the models' performance.

[00:34:13]So, for example, ETTH2 is very classic, so studies use it; it's practically a standard dataset for all studies.

[00:34:26]Beauty? So, in this case, I think it's important because, since you're in a very specific line of using LLS and also machine-developed work, sometimes you can gain something by looking for a series that has a much more complex, new statistical model that it accommodates, but it is, let's say, not very stable or it's not that interesting. Because the model has become too complex. So sometimes, when you jump to a slightly more complex time series, different from those that have been analyzed, it can help you in this aspect. There's a part you included here about the inference strategies you described; each configuration was executed five times with a default temperature of 0.7.

[00:35:24]Then I was left wondering what, what is this standard temperature of 7? It is that which is, it is not, it is that standard temperature.

[00:35:38]Well, because we can configure this temperature, right, to make the model either more deterministic or more generalist, right?

[00:35:48]So, libraries, for example, the UPI library, are configured with 0.7 by default. And I also think that in LM Studio the default temperature is 0.7.

[00:36:02]So, it just means that I didn't make any kind of temperature change to do any kind of analysis in relation to whether I have a lower temperature or a higher temperature. I simply don't change the temperature; I leave it at the default setting.

[00:36:17]So in this case, it's good for you to say what that is, because standard temperature, as you 've stated, is something very specific, a nomenclature used in the field for certain things that the reader may not be familiar with, right?

[00:36:34]And this probably isn't a temperature; this is a parameter that's called a temperature because I don't know if there's some kind of cooling or algorithm that makes you call this parameter a temperature.

[00:36:47]So, let's try to specify or put it in a slightly different way, but make it clear to the reader that the temperature appears in a way that a person glances at it and doesn't really understand what it is, right?

[00:37:10]So, another thing regarding the execution environment, I was wondering, since you put the configurations here, but it's because some things are implicit, of course, and some things are explicit, so everything you ran, you ran, or rather, up to a point in the execution of the expedited process, you put it here, so there are two points, right? The first point is that the material here, you say that you've already done initial tests, but you haven't put anything in place yet.

[00:37:44]So there's a "for" or "will be" option, but considering that and this machine configuration, it means everything was done locally.

[00:37:55]Was anything done differently?

[00:37:58]You plan to do everything locally, how does that part work?

[00:38:04]Well, I plan to do everything locally using the lab machine.

[00:38:10]Beauty?

[00:38:14]Another thing that I think is also interesting, right, even in relation to the modeling, is perhaps to leave a little more in relation to the objective of alignment, right, because the main objective with regard to these models is, you are very focused on forecasting, so there isn't a great focus on anything else besides forecasting, right? The usual statistical models, such as ARIMA, and other dynamic models are here; these are models that have a focus—you can focus on prediction—but for good use, you have to have the model's hypotheses guaranteed. It reflects that diagnostic analysis when you're thinking of an automated alternative, right? That's why you'll compare it to models that already jump, like Altarima and Alet ETS, which are automated models. So, I think it 's important to even put this in the work, to say that the objective, because a person might arrive, Altarima, it won't always release the best model, it will release the best model based on a pre-established criterion that you put there, for example, it's the one with the lowest level of analyzed information. Then the autocracy will exalt that model. I'm not worried about whether one thing or another happens to my waste. So, who are they then, or will the model release the best model, or the model that minimizes the mean squared prediction error? So that's the goal, right?

[00:39:57]These are automated procedures. So it's important to put that there, right, in the comparisons, to avoid even any reference, any proposal that suggests that if someone analyzes it more carefully, they might feel, "I'm going to take this here, but I'm going to compare it, I'm going to make a diagnosis, I'm going to look for tools, I'm going to look for other things," but since the focus is on minimizing those quantities that you put there as criteria, for example, this can be quite clear. Another point to consider for the future is that I'm treating you like a graduate student, okay? So it was just a matter of checking which models you're working with and which automatic criteria use those improvement criteria as estimation criteria. Because one thing, I chose an Altaima model programmed to give me the lowest possible training criteria. Another thing is to choose a model that minimizes those errors you mentioned, like the one in the selection process. So these models that automatically minimize performance will be better. Those models that you're using to automatically minimize something else will give a different result. This is just to give you an idea of ​​what each context is doing.

[00:41:22]Because if all the models are too free, either you're using them to minimize the distances you want to compare in the end. Did you understand it that way?

[00:41:36]So that's what I think is important, paying special attention to what's going on.

[00:41:42]So I think I've spent enough time here already, but once again, congratulations to you and your advisors. It's an excellent piece of work, and even for a dissertation it's very good. So I'll hand the floor back to Wellington.

[00:42:03]Thank you, Rafael, for your input. I found your comments to be very good. I think these issues really need to be raised. You highlighted the main difference; it's these little details that really make the difference when submitting an article to sell your idea. So, thank you very much. He's going to commit to taking all of these considerations into account, right? And in TCC2 we'll check if it's okay.

[00:42:32]Estimated there. It's OK. Thank you very much, professor. Now I would like to invite the second evaluator of the ES project, Professor Bruno. Professor Bruno, thank you so much for accepting the invitation, right? Do you want my partner to use it? Feel free to leave your comments on ES's work.

[00:42:51]Well, good morning everyone, right? First, I wanted to thank you for the invitation to participate in Wesley's thesis defense. He 's a student I've known for a long time; I think as soon as I arrived on campus, Wesley was one of my first students, in my first machine learning course. It's a pleasure to see his evolution here, and I really agree with Professor Rafael.

[00:43:14]We're going to treat Wesley as a postgraduate student, because the objective of this defense is precisely to bring some contributions to the work, with a view to possible publication, and especially considering how interesting this work is. So, I made a few notes here, right? I took notes, Wesley, right? Uh, in the PDF, right? And here are some excerpts, you know, textual ones here to more or less guide my speech, but I'll pass everything on to you, right? And right off the bat, I congratulate you, I congratulate Professor Caminha, and I congratulate Professor Wellington for the good work. It's a very interesting piece of work, it has a very good rationale, a very strong appeal, related to time series forecasting, using a new approach, right? But I have a few points here that I'd like to highlight, right? And in case you find them interesting, right? I'm even going to pull this over a little bit so I can uh uh get a better sense of direction, you know?

[00:44:25]Uh, so, uh, once again, right? I really liked the work; it has a very interesting thematic area, which could easily be turned into a publication, especially for a conference or a journal, if the scope were broadened a bit, in terms of results and evaluation. So, you know, my goal once again is to bring up some points to improve even further, and to look forward even more to your TCC2 defense, which will certainly result in a very interesting piece of work. So I separated some points here, following the chapters, based on some suggestions I found in your text.

[00:45:07]Okay, let's go.

[00:45:09]Regarding the introduction, right, at the beginning, you mention that the area of time series forecasting is very relevant, and you even justify this in your presentation. And then, commenting a little on your presentation, you present yourself well; you can see that you have a fluency when discussing the subject, and that 's typical of the immersion you always do when you're studying certain content, especially since you've always been a good student.

[00:45:38]student, okay? Well, you mention that the forecasting area is very relevant and justifies it. If you could put that there for me, Wesley, in the part of your slide, in the introduction where you talk about Covid, if I'm not mistaken, you even gave a justification about the evolution of cases in a certain period, but that's not in the text, is it?

[00:46:06]So I didn't find that part you mentioned in the text, the one about motivation, right? So it's interesting for you to bring up topics in the presentation, especially those that are in the text. Since I couldn't find this one, I suggest you add it to the text because it 's a good motivation, okay? Well, besides that, you mention that it has diverse impacts on society, you say, in agriculture, retail, energy, there are quite a few applications of time series forecasting, but it seemed a little too generic to me, right? You could, for example, explain the impact that a time series forecast can have in a given context, right? Yeah, I even brought two references here. Hey, here's a reference, let me see. She says, "Look, applying time series forecasting can, for example, reduce errors by 20 to 50%, right?" And often this can translate into impacts of 1.6 million, right, when you're making a forecast in the energy sector, which is one of the topics, right, one of the datasets you use, right, Wesley? So, if you bring this up in the introduction, the reader will glance at it and think, "Wow, this area that Wesley is addressing is really important." And it can, you know, it can bring significant impacts, right? So, what do we generally mean by significant impact? It's about money, right? It's a cost for a company, right? Imagine if I could reduce a company's costs by £1,600,000 per year, you understand? Only with time series forecasting, right, with the use of time series. So, it brings, you see, it brings a very great force.

[00:48:05]So, I've separated two references here, one isn't that recent, the other is more or less recent, right? But both of them are good references that you can add to your final version of the text, which will give it more strength, because I felt something was missing, you know? You said it has an impact, but you didn't specify that impact to provide motivation for such a rich endeavor, which is what you're proposing here, right? Well, in that sense, you know, I also felt something was missing in the introduction, a slightly better description of the gap, right? So the first question I wanted to ask you is, why is a library interesting?

[00:48:51]Why propose a legal library?

[00:48:53]It's interesting in a scientific context.

[00:48:57]Well, the library has a very peculiar characteristic, which is the ease with which we can integrate other tools with it.

[00:49:06]So, for example, in a context where we want to do a benchmark, we might also be interested in, for example, doing a computational cost analysis or, for example, the carbon footprint, right? How much CO2 will be emitted, according to an inference from the model? So, the library has a very easy way of integrating other libraries to really complement the researcher's analysis. So, it's because of that possibility.

[00:49:39]Very good. Ready. Yes, you gave a very good answer, right? It could be included in the text because it will make it easier not only for me to do benchmarks and comparisons in the literature, but it will also prevent me from having to rework an entire pipeline from scratch in a given solution. No, I'm going to use Wesley's library here, the one he's proposing, because it's already ready, right? I can use it and it can also serve as a comparison with other methods using the library itself. So, this could be emphasized a little more in the introduction, which is the purpose of this chapter, okay?

[00:50:19]Hey, let me check this out.

[00:50:25]So, thinking about the article, at the very end of the introduction, it's a suggestion here, more related to writing itself, right? I would highlight the main contributions of your work, you know? I think this will become interesting and more prominent in the second thesis.

[00:50:41]Obviously you're going to get results, you're going to deliver results, right? Uh, I think you could bring up and list the main contributions at the very end of the introduction, okay? Before the objectives.

[00:50:53]Okay, fine. I liked the overall objective; it turned out really well, but I would change the specific objectives and add a few points. So I wanted you guys to rate it, right?

[00:51:06]Eh, eh, it all makes sense later, okay? I would set, for example, a specific objective: to design and implement a modular library, which is a module. You'll have to work with it in modules, composed of the modules you brought up in the methodology. Later on, in the methodology, I'll explain here that I think I would work on these modules a little better later on, but I would explain that it's a modular library that has the vision of providing this complete composition of a pipeline, from ingestion to predictions, okay? So I think I would put that specific objective after implementing and evaluating inference strategies based on promptis engineering, which I think is one of the objectives that you put in your methodology, which could be clearer in the specific objectives, right? And finally, I think I would add, besides those you mentioned, which are three if I'm not mistaken, making the library publicly available. That's your intention, right?

[00:52:20]Yes.

[00:52:20]Make it available. Ready.

[00:52:22]Yes.

[00:52:23]With the documentation, right, to allow precisely, right, a complete reproduction of the experiments and their widespread use in the literature. Legal. Okay, now that's about introductions. Let's talk a little bit about the theoretical foundation now, shall we? Well, it's a very robust and extensive theoretical foundation, isn't it? There's a lot going on, is n't there? Yes, and it was noticeable that you perhaps started with that, right, and talked a lot about the statistical part, brought up a lot of text, a lot of figures, a lot of points, right?

[00:52:57]I don't have that many suggestions, do I? But there are a few things, for example, that I think you forgot to mention in the rationale, like talking about the temperature, right? What exactly is temperature? Professor Rafael even brought up this question here, which could be linked to the rationale. You explain this in the rationale, in the methodology you can say that everything's fine, right? But those are just details, those kinds of things happen, there's no problem at all, right? Well, we have this kind of forgetfulness, right, that we use in a way that's closer to home, right? Sometimes it goes unnoticed. So I suggest reviewing and finalizing what's missing in the methodology, specifically detailing it in your theoretical framework, because that's the objective of this chapter, okay? Um, let me take a look at some very, very small writing details here, right? For example, when we're explaining variables in an equation, we say, "a, such and such where m," and then you just change that "where," because we usually use "where," which comes from English, but we usually change it to "in which," but you 'll see, it's a silly thing that doesn't require much work to modify.

[00:54:22]Hey, let me check this out.

[00:54:26]Okay, fine. There are some pictures, right? Well, there are some tables that you present in the text, but you don't comment on them, you don't discuss them.

[00:54:37]You see, table one illustrates, right, presents, illustrates the related works, for example, let's guess here that it's related works, right? And then, right below it, you make a comment about the table. It's true that the table is often self-explanatory, right? But it would be helpful, it would be interesting if you included at least a short paragraph explaining about it, right?

[00:55:00]So that she's not just left there loose and merely presented, okay? Generally, we include the image, the table, discuss each one, and then present it, okay? Hey, let me see here, let me see.

[00:55:19]OK. the statistical models section.

[00:55:21]All good? Ready. Ready. Let's move on to related tasks, shall we? I really liked the table, didn't I? Col, could you please put it on the related assignments slide for me?

[00:55:32]You brought it ready-made, look. That's pretty interesting, isn't it?

[00:55:39]Full support, partial support, not supported. But imagine you putting this last little line here, just to position your library in relation to what you have, okay? So, uh, I think you touch on this a bit in the text, but I think putting it in the table makes it even more, uh, interesting, right? And especially thinking about it for a publication, okay?

[00:56:05]Uh, let me see some related jobs.

[00:56:09]I think that was it.

[00:56:11]Hey, tell me something, Time Copilot, right? The one you saw, how does it work? It 's open source, she searched for all the libraries herself, right? They are all open source. How does she use the models, what is the process like, and what is the main difference between Time Compilot and your library?

[00:56:34]Ready. Yes, all the libraries are open source. Uh-huh.

[00:56:39]And the big difference between this work and the time pilot, first, is how the timeout works? He uses the LLM as an agent, as a process orchestrator.

[00:56:52]So Llisão itself will automate the pipeline.

[00:56:58]So he's going to take the time series, analyze it, and based on its characteristics, determine the best machine learning or statistical model to make the prediction. This work, however, doesn't have that intermediate layer; it's direct. So we pass the data to the LLM and the LLM makes the prediction itself, which is what we call a predictive model. We transform LLM not into an agent, but into a predictive model that will make the prediction itself. So that's it, isn't that interesting?

[00:57:32]Very, very cool. And it makes your work even richer, right? Because your work positions itself differently. So it's going to be necessary here, right? Eh, eh, Professor Witon, Professor Caminha, right? Well, I'd even add some new columns here to specify that, right?

[00:57:52]Because he's incredibly rich, okay? So, your library, Time Compilot, I do n't think it even comes close to what you intend to deliver, right?

[00:58:02]So I think that to further highlight this contribution, which is very interesting, I think this difference needs to be very well specified in the text and in the table, okay?

[00:58:16]Yeah, cool, really cool. Now let's move on to the methodology.

[00:58:22]Uh, in your diagram, right? Is it possible for you to put it in the diagram you brought there?

[00:58:28]Ready. Exactly. This is a diagram, right, that looks a bit like it could be mapped to a pipeline, right? Yeah, but I understand the part about the modules too, right? Because their idea is to create a modular library. So I would separate, for example, the data part here, into three different parts, right?

[00:58:50]You said that this part of the data has the part of reading the databases, which I would call ingestion, which is a concept from data engineering that I think can be coupled here: data management, preprocessing, and exploratory analysis. You do all of this in this little data block, right?

[00:59:11]So I would put this data block and then put three more blocks inside it, okay? To make something even more modular, right? Well, I think that would also make it easier for the reader to understand. Yeah, that would separate them too, right? I've written it down up to this point, see? Uh, I would also separate, uh, where's the part about the models with sub-modules, the local ones and the cloud-based ones, okay? Because your library will also provide that support for both of those things, right? So, whether I want to put a local LLM (Local Market Logistics) or just put her little pizza there and call it, you know, on the cloud LLM, okay? I think I would also divide and subdivide in that sense, okay?

[01:00:02]Well, overall it turned out really well, I understood that part about the modular section very well. So I wanted to ask you something about the criteria you used, you know, in the 24- step-ahead forecast horizon and a 70-day training window. What specific criteria did you use to define this?

[01:00:30]Ready. Regarding the 24 periods, right, the 24 values ​​of the forecast horizon, it's because ETTH2 is an hourly time series, so when we take 24 values, it's as if we're trying to predict, no, we want to predict a day, right, 24 hours.

[01:00:51]So, and also taking into account the seasonality of the time series, right?

[01:00:55]So, since it has a very strong daily seasonality, it was chosen for that reason and because it's more conventional, right? 24 periods is a day. And the 70- day context window was chosen because it 's a middle ground between what I can actually pass in the prompt, because these models, these LLMs, have a limited context, a limited context size. So I can't pass all the data at once, but I also can't pass only a small amount of data.

[01:01:34]Because the model won't be able to provide enough context to make a good prediction. So, 70 days is a very good balance, it's manageable.

[01:01:45]Pretty cool, huh? But this is directly linked to you and your experiments, right? Specifically, the database you're evaluating, isn't that right?

[01:01:55]The databases you're using are for comparison, right?

[01:02:02]Sorry, the internet went down, can you repeat that? Okay, so, this 24-hour thing, right? It's totally related to your database, the databases you're evaluating in your work, right?

[01:02:16]Well, regarding the context window, right? If you have, if you can find a reference that provides a little more support, all the better, so it does n't remain so empirical, that would be interesting. If you don't agree, I would vary it; this would be part of what I'm going to comment on in a little while, which was something that worried me a bit regarding your experiments, okay? Hey, let me see if you've included table number eight. Did you put it there?

[01:02:54]This is a table that contains the number of estimators; it's a table that has some parameters that you've set fixed.

[01:03:03]Yes, it's not on the slides, but it's not there, right? Let me, let me see here just so I can... Can you share it there, just you can find one... Can I just...? Ready. Put it in table eight.

[01:03:19]Share your screen and then upload it to the table, or you can upload a PDF.

[01:03:26]Okay, now you can see it.

[01:03:28]Perfect. You can zoom in there.

[01:03:32]Ready. Oh, uh, for all these models you're comparing, some are based on more statistical data, right?

[01:03:40]Well, some are based on deep learning, others on trees, and others on temporal neural networks, convolutional temporal networks.

[01:03:51]Well, that's the main point I wanted to suggest here, okay? Well, to make the configuration of these models standard, you know? I don't know how long it would take to run that battery of tests, but using that default configuration would limit your work, you know? And I would change that entire default configuration approach to grid search. I would use a grid search or a random search, in the case of algorithms based on deep learning, to facilitate the search for the best hyperparameters for each of these models, okay? Hey, you're saying here, "Why is the number of estimators 100?" What was the criterion for choosing him, right? So, I believe it has to be varied.

[01:04:42]10, 100, 1000, right? Varying by 50s.

[01:04:45]This is very empirical, okay? And I wouldn't change it, and I would even add that, like grid search, the context window also varies, varying the context window, right, by 30, 40, in short, in a quantity that you could run to see how the models handle it, okay? the different models of different families that you're evaluating here. So I think that was another really cool contribution from you, bringing robustness to your methodology related to this point, you know, Professor Bruno, but just so I can say, I liked your suggestion, but in this case, the idea of ​​his work is a baseline, that's why we put the standard there, right? If we're going to vary things, then we also have to vary the LLMs, right? You have to vary the input parameters, the points, the context, right? Well, that would definitely result in a more robust piece of work, wouldn't it? In the sense of comparison, of experimentation, right? But uh uh, that's the idea here, right? I think initially it's like this: let's take a pattern like this to see if it's competitive, to see if it makes sense, you know? He knows? His library, right? It's competitive, uh, it can enter these algorithms and run and compare them, you understand? And uh, but I understand, uh, uh, it gives some flexibility, a way to change the parameters and try to bring some result already within its library, right?

[01:06:29]Yes.

[01:06:29]It is, especially because I understand the idea perfectly, right? And here we are again, right? We're treating Wesley like he's a graduate student, right?

[01:06:37]For sure. For sure. [laughter] Hey, rat up there.

[01:06:42]Yes, but the idea is precisely to imagine that within your library there's a function that creates this grid here, you understand? I think that would give an absurd amount of profit, wouldn't it? That's even more than you already plan, right? Yeah, you have to, right?

[01:07:00]But I totally let it go, right? Just a suggestion, don't worry, this isn't something very critical in my opinion, okay? But I would include it to, you know, draw attention to the results and say: "Look, I'll show you which is the best model and I'll also tell you what the best hyperparameters related to this model were, okay?

[01:07:24]So, maybe it would be interesting, but let's see, right? Let's see how much time he'll have to finish his TCC2, right? From now until December, let's see what he can develop.

[01:07:36]Okay, let me see here.

[01:07:44]Finally, my friend, I felt something simpler was missing. So, as a suggestion, another suggestion would be to statistically validate your experiments. How so? We can use, Professor Rafael Faris can even correct me here, it's not exactly my specific area, but you're evaluating, right, the mean squared error, the MAP, right, and some other metrics, right, across these models, different models, right, within this battery of tests that you do, we could statistically compare them." The metrics are important to see if we really have a difference.

[01:08:25]Statistics, and not just, oh no, the mean and standard deviation gave this value, and it's just a number that won't help me make a decision, right? Uh, uh, about which model to use. So, depending on the size, the number of repetitions you use, uh, I would use, for example, the WCXson test, which is non-parametric, and it could work with these smaller samples, since they don't follow, obviously, because they are very small, I can't say they follow a normal distribution, right? So I would also include a statistical test within the metrics you use to evaluate these models, which would further highlight the contribution of your work and prepare it for publication in a very straightforward way, okay? And that's all. So, once again, congratulations to Wesley, congratulations to Professor Caminha and Professor Wellington for the excellent work, and we are available if you need to contact me, ask questions. Any questions, we're at your service, okay?

[01:09:32]Hey Bruno, thank you so much for your revisions. It helped a lot, man. I think it's a 73- page work by Wer, right? It shows that it has potential, that his work has potential, and we 're going to take all your suggestions and make the work really polished, right? I think we can, well, he has a publication, right, but I think we can really publish it with these suggestions and make it a more robust and better work, right?

[01:10:09]So, thank you very much, Bruno.

[01:10:11]Now, without going on too long, I wanted to thank Professor Caminha for being here, who is the commentator for ES.

[01:10:18]Thank you very much, Professor Caminha, for this partnership, both in relation to ES's work and other projects that we already have for other students, right? So, open the microphone here, feel free, professor, to put your suggestions, contributions, comments on the work of... Oh, thank you. Can you hear me well? Can you hear me well?

[01:10:41]Perfectly, perfectly.

[01:10:42]Okay, okay. Good morning again, everyone.

[01:10:44]Uh, uh, I wanted to say hello, the entry was a bit chaotic due to audio problems. I didn't greet Professor Bruno or Professor Rafael. Thank you very much for your comments. I wrote everything down, I'll share it here along with the recording, even though it's being recorded, right? I imagine there won't be a problem retrieving everything later. Uh, thank you very much for your contributions. I wanted to thank Wesley for his presentation. I don't have any further comments. I think what I had to say about this work, I've already been saying in meetings for a long time. And I just wanted to congratulate Wesley, uh, a TCC1 (Final Course Project) with a publication in a conference within ICLR, right, in a workshop within ICLR, a top conference worldwide. And Wesley is there as the first author of the article.

[01:11:39]Putting this work here... It speaks to the magnitude of what this is, right? What... what's in these 73 pages, right, of a TCC1... uh... uh... to show you the potential that's here. So, I'm very happy. I also wanted to mention here that Wesley's work is part of a larger project done within the laboratory that Professor Wip and I coordinate. It's called Leb UFC, it's a partnership with the Konome Institute, there are almost 40 scholarship recipients working on it. Wesley is one of them and he has stood out a lot in this area of time series forecasting using language models, along with Zairo, Fernanda, there are several other scholarship recipients, in fact, Crateú is... Crateú is... is having a very large impact on the project. We also have Gustavo, several other students, Micael, Marlon, several who have left and then, right, and others who are concluding... And it even raises... Questions like, what's in the water from Crateus, right? What are they giving these kids to drink? What's going on?

[01:12:48]Because, of course, it's a game that's the result of his work and the teachers' work, which is what's truly special. So, I just want to congratulate everyone, express my satisfaction and pride in being with you all here in these initiatives, and congratulations, congratulations to everyone, and thank you Bruno and Rafael for the comments. I'm sure we 'll meet everyone's needs and make TCC2 even better.

[01:13:17]Thank you. Thank you, Caminha. And, thank you again for the partnership. So, folks, when I get back, I'll talk a little about Wer's work, right?

[01:13:27]As usual, the panel will leave, right? Go to the deliberation room.

[01:13:32]We'll be back soon to give the results. Go back to the first slide, while we go out.

[01:13:42]And then, take the opportunity to comment with your audience... Okay? So we're going to step out here, take a look at the messages, lots of people are watching, right? And we'll be right back to give the verdict, OK?

[01:15:22]Oh, now you can hear it. M.

[01:15:39][laughs] Fernanda's aunt. A kiss to everyone who's watching.

[01:15:59]Thanks, Miguel, thanks, PK.

[01:16:04]Only the best here.

[01:16:11]A kiss to all the people from Caucaia, the people who are watching.

[01:16:18]Dagoberto.

[01:16:30][laughs] A kiss to Canã do Carajaço.

[01:16:45]Shout out to Miguel. That's it.

[01:16:59]A shout out to Tainer, who has a very different name, right?

[01:17:14][laughs] Okay, folks, coming back here, right? Uh, uh, can you hear me well?

[01:17:28]Yes, I'm sleeping perfectly.

[01:17:31]So, folks, uh, before giving the verdict, Wesley, uh, the panel discussed, butted heads a lot here, right? Even wrongly, because we discussed so much that it didn't take long. Uh, Okay, so before giving the verdict, I wanted to thank our host, Professor Rafael, vice-coordinator of Computer Science, for being here accompanying our broadcast, handling all the technical aspects. Thank you, Rafael. And now, I'll give you the result.

[01:18:08]Congratulations, you passed TCC1.

[01:18:11]Congratulations, closing one cycle, right? And for the next semester, you'll only have TCC2 to complete your degree in Computer Science. I just want to emphasize, Weser, that your work was a work of excellence within the Instituto Conalho, a partnership we have. So, congratulations on your work. And as Professor Rafael and Professor Bruno said, now we expect something better in TCC2, you know? You raised the bar, so we have many suggestions. I'd like to thank Professor Rafael again for the suggestions, and Professor Bruno and Professor Caminha for their suggestions as well. And then... We're going to work hard to do everything that people asked for, do more experiments, deliver an even better TCC2 (Final Course Project 2), and work towards your second publication in TCC2. Okay? Okay. Now I'll open it up for you to say something if you want, feel free to give your speech.

[01:19:26]Well, I'd like to thank the entire committee, especially my advisors, Professor Wellington, Professor Carlos Caminha, thank my whole family and also my friends, my research colleagues, Záiro and Fernanda. And everyone who directly or indirectly contributed to me being here. So, thank you all very much. That's it.

[01:19:51]Okay. Congratulations, Wesley, on your achievement!

[01:19:55]And that's it, much success with TCC2.

[01:19:58]Let's get going, there's still a semester of hard work ahead, right? If anyone from the committee wants to speak, feel free to comment.

[01:20:10]Congratulations! Thank you all.

[01:20:13]Congratulations as well. Congratulations. Success.

[01:20:17]Congratulations, Wesley. And hey, we're at your disposal, okay? Anything, talk to us so we can move forward even more and count on us in the research, okay?

[01:20:28]So folks, let's wrap it up here, right?

[01:20:30]Thank you again to the course coordination, to everyone involved in the broadcast, to all our viewers, right, internet users who are following the ESER defense.

[01:20:43]That concludes another TCC1 defense; soon there will be other defenses for you to follow. Thank you. Have a great day and see you later, everyone. Thanks. See you later. Hugs.

[01:20:53]Ciao.