TUG 2026

[00:04:40]It it is showtime. Yes.

[00:04:46]So, just have a very short announcement concerning the photographs on the back wall there. If you haven't looked at them yet and you've been to previous meetings, it'll be a treat through history for you. So, these have been put up to be enjoyed, but as a bonus, at the end of this meeting, if you would like to have one as a souvenir, and it doesn't have to have your own picture in it, you can have it as a souvenir to take home. And it's all for you.

[00:05:18]>> So, yes, >> doesn't have to be one. Take more than one.

[00:05:21]>> It does not have to be one. in LPG it would be.

[00:05:25]>> Yes.

[00:05:26]>> Zero.

[00:05:27]>> Yes. Okay.

[00:05:30]>> So, incidentally, I'm supposed to say who I am. So, I'm Michael Doo. I'm pleased to be here at this tug meeting.

[00:05:38]I'm not sure how many I've been to, but it's definitely in double digits. So, and I must say this one is, as with so many of the others, has been just a true delight. And I would like to thank the organizers who have put this together.

[00:05:55][applause] We we are here on the last day and it's been wonderful so far. I'm sure it will continue. And don't forget there's a dinner tonight. So for people who are watching online, sorry, it's too bad.

[00:06:16]But you'll have to come to the next TUD meeting in Paris. So, okay. So, um our timing is pretty good.

[00:06:26]Here's our program for this morning. Uh we will start in two minutes, I guess. So, I'll give you a chance to finish up your coffee and we'll be able to get going.

[00:06:41]>> The photographs are also available on the credentials and whatnot.org.

[00:06:49]>> Uhhuh. Yes, for those who are here who can read the back board, you can also get the photos online.

[00:06:59][snorts] >> So, you know, some people will be logging in from the outside, so I don't want to start till, you know, nine o'clock. We got a little off schedule yesterday, and I it sort of concerned me a little bit.

[00:08:08]Front left.

[00:08:14]Front left. Front right.

[00:08:48]front left.

[00:09:27]And then >> okay, you'll leave this on for while I make the announcement. Is that okay?

[00:09:34]>> Oh, okay. That's good.

[00:09:39]>> Uh, okay. So it's time to start. So we'll have our first talk of the day. So uh our first presenters are KB Rajish and CB Radah Krishnan and they will be talking on Malayam content 2.

[00:10:06]Hello, welcome to the tech 2026 presentation about digitally archiving Mallayalam text. This is a sequel to the talk delivered by CVR at tech 2025 last year. Since then we have considerably reworked the workflow to make it simpler and more efficient. I would like to provide an update to the workflow in this talk.

[00:10:32]The previous workflow was a bit contrived. It involves using Lua deck to produce a special kind of PDF which is not suitable for immediate user consumption because it only contains the hierarchically structured XML board. From this PDF, the text was extracted and transformed to produce an XML document which involves quite a bit of manual steps as well as expertise.

[00:11:02]This workflow was developed more than 15 years ago and then tech for STD was not capable of processing uni code text for complex splits. But in the last few years thanks to the UI tech and Z tech engines as well as make ofix works. Tech for SD is now fully capable of supporting uni code text as well as complex scripts like malar.

[00:11:28]So we wanted to make use of the native capabilities and strengths of tech forestry to process latex sources.

[00:11:37]In this workflow the the immediate output is in HTML not XML. The difficulty in creating validated strict XML directly from latex multiffold. First of all, tech is quite macroheavy and authors may use all kinds of complicated macros.

[00:11:56]And secondly, latte output is visual output suitable for printing or reading on screen. It does not generate a structured data. XML being structured data, creating such kind of output directly from the tech is quite complicated and at times errorprone.

[00:12:18]Crucially, our volunteers who are also less skilled in XML technologies at times struggle with fixing the parse error in XML or its structure.

[00:12:30]So we wanted to separate the concerns of the lesser knowledgeable workforce volunteer workflows at Sana Foundation to work with lighted sources. They can compile the sources. they can fix the compilation and errors and the immediate output is in HTML which gives immediate visual feedback so that the volunteer can check the output verify and correct the [clears throat] sources if needed and repeat the cycle. Once the HTML is verified then this adds limited number of expert volunteers can then work on the next part which is to produce XML from this HTML.

[00:13:12]So the new workflow involves compiling LE sources directly using techd which produces a semantic HTML.

[00:13:21]This semantic HTML can then be converted to X HTML which is similar to XML.

[00:13:29]From this X HTML any kind of structured schema XML can be generated using XML processing tools such as XST or DOM or SX etc. This helps us to separate the concerns of um limited expertise versus comp complexity.

[00:13:55]HTML 5 is quite suitable for our workflow. Firstly, because it provides immediate visual feedback.

[00:14:02]Secondly, latex structures have equivalent mapping to XHTML. Semantic HTML for example / section can be directly mapped to [clears throat] the section tag or text it for instance can be directly translated to the I tag or a span tag with class italic which can then use a CSS for rendering. Most importantly tech can generate math ML directly from latex sources. Tech for SD has been in our production use since many years. It's battle tested and it has excellent support for standard Latic packages and macros.

[00:14:42]In addition, the HTML output is also quite forgiving because if you miss uh to close a tag, it could still be rendered correctly by web browsers. But if you miss to close a tag in XML, it's a parser.

[00:14:57]It's very easy to convert semantic HTML into XHTML. Once we have the XHTML, then any standard XML tool chain should be able to produce a required XML schema.

[00:15:13]Tech forests works in three independent but coordinated steps. First step is to compile the sources using tech for SD which loads the tech4d.sty sty package that redefineses most of the macros and in its place it injects special anchors to the DVI output. Once the compilation is over these special anchors can be extracted from DVI and replace it with XML, HTML or MathML tags.

[00:15:44]The final stage is run only if you need to extract graphics or if you need to split the output file by chapter sections etc. The real advantage of tech using tech for is that it uses a real tech engine such as PDF tech.

[00:16:06]So it process all the macros as if a PDF output is being generated.

[00:16:14]Tech for STD can convert standard lat commands to equivalent semantic HTML.

[00:16:20]Sometimes if you need to customize the output, the way to do it is to use the configure command. For example, let's convert the /se section command and latex to an equivalent semantic HTML section tag.

[00:16:35]Use the configure section command which [clears throat] takes four arguments.

[00:16:39]The first two Hboard code section level two and the Hboard closing tag section can be used to add specified port just before the start of the section block and after the entire section content is over. The last two arguments here in here in the example H code S2 and closing H2 tags can be used to add XML or HTML tags before the section title and after the section title.

[00:17:19]Once tech for ST runs with this configuration, it will produce the equivalent output that you you can see on the right hand side.

[00:17:31]Mathematl generation is [clears throat] straightforward with tech for ST. For example, if we have an inline math equation A + B the whole square, simply compile it using make for ST with the option math ML. It will produce Math ML output as displayed below including the math class for the operators and delimiters.

[00:18:04]The second challenge we had with uh digitizing Mallayon text at Sana Foundation is we have volunteers with varying levels of expertise.

[00:18:17]The lesser skilled volunteers find it difficult to work with XML and fix its errors. Instead they work with latex sources which they are quite familiar with and they standardize the coding into strict predefined templates so that we use equivalent semantic environments not just formatting commands such as space or text pf and using tech for HD the sources are compiled into HTML from which they can get immediate visual ual feedback and correct resources if necessary. This process is repeated until the generated HTML is satisfactory and correct.

[00:19:03]Once that HTML is created, then the SK expert volunteers can focus on the XML generation part and fix any errors.

[00:19:12]And this is where the domain experts like linguist also come into picture where they can do named entity recognition linking critical apparatus or XML schema validation etc. This um decouples the processing stage into varying levels of expertise and relieves the expert volunteers from the mundane tasks.

[00:19:41]We have also built an automated pipeline which is basically uh make file that bundles three different steps. First step is to compile the input source file using tech for ST with the option MML X HTML and HTML 5. This produces an HTML 5 document which needs to be converted into XHTML with proper opening and closing tags.

[00:20:10]For this purpose, we use the command line utility tidy.

[00:20:15]Once tidy runs through this input HTML, we get a strict X HTML which can then be transformed into desired XML schema such as TEIXML or NLM objects by using Excel prog or Saxon or other tools.

[00:20:34]We have a case study of Kerala paninium an authoritative mallayalam grammar text published by sina foundation few years ago in PDF format in the last year. We also worked to generate the HTML and XML format of it using the new workflow.

[00:20:57]We have also [snorts] processed a number of uh articles, books, poetry, etc. using the new new workflow and those are available on the SA website.

[00:21:12][clears throat] One particular stage of um XML generation has to do with the domain expertise. For example, if you have if we have um an archival document which requires enriching with additional semantic information such as named entity recognition and linking or critical apparatus for example the apparatus lema uh and reading this this can be added during the HTML to XML conversion process. In addition, strict schema validation can also be automatically introduced, for example, using XML link.

[00:21:54]So, we automate it as much as possible, but the meaningful part is still left with the experts and domain experts.

[00:22:05]Compare the old workflow with the new.

[00:22:09]Previous workflow has been using luate tech to create a specialized PDF which is completely removed to use HTML produced by tech for HD and in the and in the previous workflow all these steps require high level of skill but now we have separated the first part of generating HTML from related sources can be run by much lesser skilled volunteers.

[00:22:36]uh the XML generation was also a manual process in the previous workflow which is completely automated using accessibility processing in the new workflow and the previous workflow did not support MathML. So we were unable to create XML and HTML of the 15th century mathematical treaty called Yudi Bhasha which Sana Foundation published few years ago in PDF format. But with the new capabilities, we will be able to to generate the XML and HTML version too.

[00:23:10]Since the previous workflow was complicated and contrived, uh only few people were able to work with it. But with the new workflow, a lot more volunteers can work with it.

[00:23:25]The major challenges implementing the new workflow was the initial configuration of tech for SD. An example was sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh shown earlier to configure the section command. But it is much f much easier than writing lat macros for instance.

[00:23:41]And uh creating XSLT uh programs to ensure strict HTML generation was the initial challenge and balancing this complexity of the configuration versus the lesser knowledgeable volunteers who work with latex sources to strictly use specified templates was another challenge. What we learned is that uh limited set of letter commands and strict semantic environments using a letter template rate greatly helped and HTML 5 is quite robust forgiving and few human readable close to XML format that could act as a bridge between LE sources and XML and um lesser knowledgeable volunt are familiar with latic and they're familiar with latic errors. They know how to fix those errors. So let them fail gracefully in that stage is much easier and productive than let them also fix the generated XML errors.

[00:24:50]This whole project is open source and it is driven by the community. Source code is available in GitLab under the organization SANA. Project name is archiving.

[00:25:01]All the code and documentation is licensed under latte project license or equivalent free software license. The documentation is also included in the repository which helps anybody interested to chart.

[00:25:19]This workflow is language agnostic. So it could be extended for other Indian languages languages like Tamil, Telugu, Canada or other index languages like Hindi.

[00:25:33]With that I would like to thank the organizers of the Tech Users Group Conference 2026 and I welcome any questions or discussions.

[00:25:44]Thank you very much.

[00:25:56]We seem to have no signal on here.

[00:26:01]Okay, we're on a Oh, thanks so much for that really interesting presentation. We have uh a few moments for questions if anybody has them.

[00:26:20]Yes.

[00:26:26]>> Okay. Hi, I'm Mai Kruger. It was a very interesting presentation. Thank you very much. One question I was wondering about. Did you We now have a lot of the tagging support which actually adds a lot of structured information to PDF files. Did you consider if it would now actually be possible to extract that directly as XML from the PDF and basically skip text for HT and for these kind of use cases?

[00:26:57]>> Um no Marcel we have not considered um extracting XML tags or the accessibility tags uh from PDF and using that to generate XML. Maybe it's something uh that could be considered but for us the forest works quite well.

[00:27:20]>> Thank you.

[00:27:25]>> Hi, this is Pavit. I was uh curious that when you talked about segregating the workflow for lesser skilled and more expert users, did you design the workflow around the skill set of those two populations or did you find people assuming that there was a lesserkl population of workers that you wanted to use and you were trying to find those that fit the bill?

[00:27:53]The Syana Foundation um is a volunteer-driven organization who is um involved in archiving um Malayal [clears throat] content and we have as I mentioned in the presentation a number of people who are not XML experts since it's a volunteer um workforce people come and people go uh so we had to adapt our workflow based on the availability of the of the volunteer workforce.

[00:28:25]So yeah, it's the latter that we design the workflow, redesign the the new workflow so that it matches um the skill set of the uh workforce and uh we found it also quite beneficial for the expert workforce who doesn't have to be bothered about the the nitty-gritty details for the earlier part of the processing.

[00:28:51]they can focus on the important um such as fixing XML errors or um involve um linguists or language experts um who could work on enriching the XML part.

[00:29:13]Okay.

[00:29:18]>> Hi Raj, it's Carl. Uh I just wondered how many how big the project is, how many uh lower skilled volunteers there are, how many expert volunteers there are, how many documents you've archived approximately.

[00:29:33]Just wondering.

[00:29:37]>> Okay. Um the question to the first part the number of workforce or the volunteers probably could be counted on a single hand including both the non-expert and the expert volunteers.

[00:29:51]Couple of the expert volunteers one of them is CVR the other one is Ash Kumar.

[00:29:57]Uh and then we have a few uh people who work on lat to get HTML generation and pre-processing part. Um so far we have to my know to my knowledge which could be uh observed by now we have processed more than 10,000 pages of text including the old workbook and the new workbook and we published a dictionary for malaram um called shatari which was a huge community effort um I think it was during the covid period where a lot of people were were able to find time to contribute this and the grammatical text. Plus, we also wanted to generate the XML format of Yudi Bhasha, which is a 16th century treatise on mathematics and astronomy which discussed the um fundamentals of calculus um at least a century before Newton and Leitz. We produced the PDF format but was not able to produce the HTML and XML version because of missing math ML capability.

[00:31:09]So it's a future project.

[00:31:12]um all these documents there are a few English text as well um such as um BR Ambedkar's discourses on on cast in in Indian context plus a few short stories by Moasan in addition to the Mallay content it's all available on the [clears throat] cyana.org or website.

[00:31:43]>> Okay. Well, thank you for answering those qu those questions and uh are we about ready to proceed on?

[00:31:50]>> Okay. Thank you again. [applause] >> Thank you very much.

[00:31:58]>> Is this microphone on?

[00:32:00]>> Oh, usual.

[00:32:07]just speak up.

[00:32:10]>> Yeah.

[00:32:15]>> It's all right.

[00:32:15]>> I think we should I think we should wait for 9:30.

[00:32:18]>> Yeah.

[00:32:20]>> Yeah. So, um as you see, the next talk's at 9:30. Um people may tune in particularly to see this talk. So, it would be really cruel to them to start early. So, so, so rather than to have unhappy users, we'll give you a moment to ha have a little pause or even grab a little bit of coffee at the back, but we will start promptly at 9:30.

[00:32:58]>> Did you all hear that?

[00:33:01]food outside a little bit if you want to grab a late breakfast.

[00:33:17]Oh, wait.

[00:33:28]>> So, now I'm gonna have to go out and grab everybody to get them back in at 9:29, but I guess it's okay.

[00:33:38]9:24. So, it's five minutes. It's It's good.

[00:33:42]>> I guess I'm talking to the world here.

[00:39:46]A coffee break is delightful, but all good things come to an end at some point.

[00:40:01]>> Okay, we're set to go.

[00:40:03]Okay, so we're ready for the next talk.

[00:40:07]Our presenters are Andre Matelka and Peter Syka and they're going to be talking about UTF pat.

[00:40:20]Hello everyone and warm welcome from Berno. My name is Andre Matalka and uh today I will be presenting on my behalf and on behalf of my co-author Petra uh about UTF pad game hyphenation without borders.

[00:40:39]Let me start with a brief overview of the presentation. Uh the first thing will be an introduction where I will talk about the context of the article.

[00:40:50]Then I will move to motivation part. Uh the third part is about the UTF plugin itself where I will talk about the implementation the improvements we did and the integration into the landscape.

[00:41:07]Then uh we will move to the evaluation part where we will introduce the experiment and the results we observed.

[00:41:17]And finally there will be a conclusion part where I will summarize the outcomes of the article.

[00:41:28]Hyonation is a technique used on typography uh to split words at the ends of lines. The editors use it to uh achieve nicer layout of the text with block like alignment or consistent spacing. It becomes uh even more relevant uh in paragraphs that are really tight such occur in smartphone screens. For instance, software is embedded also in text editors or web browsers. The most widely used approach to the word hyphenation problem is through hyphenation patterns.

[00:42:08]These can be generated from dictionaries by the plugin program. And the example below shows an entry from such hyphenation dictionary. This one is Mariam Webster. And the hyphenation itself can be seen as the first thing on the second line.

[00:42:31]Few words about padan which I mentioned on the slide before. Padan stands as an abbreviation for pattern generation program. The first version comes already from 1983.

[00:42:46]However, uh there were some updates and the latest version 2.4 is from 2020.

[00:42:55]It is written in web literate programming language which is a combination of the Pascal programming language and plain text documentation.

[00:43:07]The core idea of plugins algorithm is to iteratively loop through the input dictionary and to collect candidate patterns. These patterns are then evaluated with uh their frequencies and the user specified acceptance criteria.

[00:43:26]The program uses mixed command line based and file-based approach. That means all inputs and outputs are specified through files and additionally the program prompts for numeric parameters at runtime. Here you can see a high level scheme of the algorithm. In the bottom left corner there are three input files. The dictionary which contains the hyphenated words. The pattern file where an initial set of hyphenation patterns can be specified and the translate file which contains a character mapping that means the lowerase and uppercase characters. The output file in the top left corner is also provided to PDM but uh it is used as an output parameter for storing the generated patterns. We can divide the execution into three phases. In the preparation phase, the initial patterns are parsed. The character mapping from translate file is parsed and the user specifies the range of hyphenation levels. The pattern generation phase contains the dictionary loop itself. The user provides the range of pattern lengths and the acceptance weights. These uh are used to evaluate the candidate patterns and select those that will be part of the persistent set.

[00:45:03]This is repeated until all the levels were processed. The output phase then writes all the persistent patterns into the output file.

[00:45:15]The user may specify that they want to hyphenate the dictionary as well. In that case, another file called patoper is created and this contains the hyphenated dictionary.

[00:45:31]Although the algorithm is quite efficient, there are several technical limitations that make the pattern generation impossible in some cases.

[00:45:41]Firstly, patan post hard bounds on the input dictionary. First and foremost, it allows only a certain number of lowercase character to occur in the dictionary and additionally, it limits the word length to 50. The processing capabilities are limited as well with only nine hyphenation levels and patterns to the length of at most 15 to be generated. And lastly, there is some technical depth which stems from the age of the program. For instance, the modularity is quite low which makes the maintainability and testability even lower.

[00:46:29]As a consequence, PGAN is practically unusable for some use cases. Firstly, when processing data from the web, there is a chance that we will encounter more distinct characters than Pin allows.

[00:46:46]With larger input alphabets, we would be able to explore the field of universal patterns. This means uh joining several languages into one set of patterns and consequently saving some space.

[00:47:02]Secondly, we run into problems when the hyphenation dictionary contains words that are too long. This can happen for instance in German which uses compounds quite often.

[00:47:16]With these issues in mind, we started working on a new hyphenation program that we call UTF.

[00:47:24]We chose the C web language for implementation.

[00:47:27]CB is a successor to web which still follows the literate programming paradigm but switches Pascal for C. The reason is that C offers few features that we find useful. For instance, the dynamic memory allocation.

[00:47:47]We managed to implement the algorithm almost identically to plugin with just three exceptions. Uh firstly, we changed the character mapping so that uh the alphabet size is increased.

[00:48:01]Then we adjusted the weights and levels representation so that the scope of these parameters is increased as well.

[00:48:10]Finally, as mentioned before, we use dynamic memory management so that the static structure sizes of pan are no longer an issue. We also perform several general refactoring tasks such as uh variable localization, grouping of related variables, function decomposition, and we eliminated the go-to statements which are deemed harmful.

[00:48:37]Let me now talk about the character mapping which was the biggest adjustment we did to the patan algorithm.

[00:48:45]We face the challenge that the algorithm was designed to work with just single bite characters which would leave us with only 256 distinct characters in the input output.

[00:49:00]So we proposed a method that translates each character into a uniquely decodable sequence of bytes.

[00:49:10]It is a two-step procedure. Firstly, we assign each character a unique index from one to n and then we convert the indices into uh sequences.

[00:49:25]You can see an example mapping in the figure on the right. We start by assigning only single level sequences and add another bite only when the smaller ranges is fully exhausted.

[00:49:40]Thanks to this trick, we keep the mapping compact and memory efficient. We can also easily distinguish the sequence boundaries so that we are able to process only those positions that correspond to places between original characters. draw inspiration from the UTF8 character encoding which also uses variable byte lens, hence the name UTF.

[00:50:02]Again, to achieve even bigger space efficiency, we can sort the characters in the translate file by their frequencies in the dictionary so that the most frequent ones get the shortest sequences.

[00:50:17]This slide shows the compilation process of the program. The C web language allows us to generate not only the binary itself but also a PDF report that contains the formatted documentation sections. Both targets require utilities from the techl distribution. Cangle extracts the coursile and cv transforms the file into pl do document. From those points the compilation is standard for the respective languages. Here we summarize the improvements of our implementation over poden. In several aspects we remove the bounds completely. See alphabet size, input word length and input letter size. While in others such as hyphenation levels, weights and pattern length, we increase the scope significantly. Newly the user does not need to reconfigure the structure sizes if their application does not fit the default setting. This is done dynamically at runtime. We also dropped the platform dependency in the case of acceptance weights. Although it may mean that the value range is smaller now, it is still sufficient in our opinion.

[00:51:32]We strived for maximum interface compatibility with plugin. Besides the hyphenation level and weights representation, the two programs are completely interchangeable. And even this difference can be solved by a simple command line translation. TF plugin will appear in the TL distribution next year and the source code repository is already published under the MIT license on Satan and GitHub.

[00:52:00]Now the presentation moves to the evaluation part. We run udf pad gun with several configurations and observed its behavior compared to padan. As the base for experimentation, we created the hiffbench collection of hyphenation dictionaries. Hipbench comprises 18 data sets in total from 15 languages including non-Latin scripted languages such as Russian and Ukrainian which use the ciric alphabet, modern Greek and Thai. The data come from various sources of different quality. The collection contains data sets created by scraping victionary as well as polished academic dictionaries. The benchmark also comes with multiple parameter profiles. These are configurations of input parameters such as the number of levels, the pattern length and the acceptance weight. The variables we observed fall into two categories. The correctness variables check the algorithm outputs against the outputs of patan and the performance variables compare the two programs in terms of execution time and memory consumption.

[00:53:17]The results were rather satisfying.

[00:53:21]The test did not exploit any discrepancies between the implementations and we were able to process a joint bench dictionary with over 300 characters. The performance measures dropped as expected but the relative change was acceptable. Both execution times and consumed memory exhibited less than three times worse over PGAN which we deem an okay trade-off for UTF Pan's flexibility. It is important to note that the parameter profiles played an important role in the observed ratios as well. This final slide concludes the presentation. We introduced UTF pad a program for hyphenation without borders. UTF Padgen extends the original Padgen program in several ways. Thanks to improved character mapping, we are able to process dictionaries with arbitrarily sized alphabets. Further adjustments to the memory management and parameter representation allowed us to increase the flexibility of the program. And lastly, our implementation underwent a refactoring phase for better testability and maintainability.

[00:54:38]We evaluated UTF pad gun on the hiff bench collection of hibernation data sets. We did not find any algorithm floors and observed only a modest performance drop compared to plugin. The integration into the open source landscape is ongoing with the source code repository freely accessible and the program waiting for inclusion in tech 2027.

[00:55:05]And that was all from me. Thanks for listening and have a nice rest of the conference.

[00:55:16][applause] >> Oh, thanks so much for this uh nice talk. Oh, the microphone's working again. I guess this a microphone with artificial intelligence. Who would have believed it? Uh it it decides that I'm okay now. Thanks so much for that fine talk. We have a few moments for questions if uh we have any from the audience.

[00:55:45]>> Since you are accepting uni sorry I am I must say ka since you're accepting uni code input what is your definition of a character in your character splitting?

[00:56:02]Um do you mean character in the input text?

[00:56:07]>> Yes.

[00:56:09]>> Uh just yeah uh as it's mentioned it can be an unicard uni code character and if I'm not mistaken uh yeah it should be just UTF UTF8 encoded character.

[00:56:31]So a unique code point basically.

[00:56:35]>> Yes.

[00:56:36]>> Yeah.

[00:56:36]>> Yes. It is >> similarly as ZTE accepts UTF8 inputs. Uh UTF accepts any UTF8 character.

[00:56:50]Did you consider if how it would affect the pattern generation if instead of looking at uni code points you would look at graphim clusters to avoid having these uh separate characters for combining stuff?

[00:57:05]>> It seems that Max is speaking but we do not hear anything.

[00:57:18]Oh, I I didn't hear the question, but could you please repeat it once more?

[00:57:25]>> Sure. Uh, I was wondering how pattern generation would change if instead of uni-ode code point, graphim clusters would be considered as input units.

[00:57:38]>> What was the what was the thing that should be instead of UTF8?

[00:57:45]um uni code graphim clusters so that you combine combining characters I'm not so proficient in how how this this works on the the bite level but I guess I guess as long as these uh clusters are not hyphenated anywhere in the in the data like there is no hypen between between those those three parts or two parts of of the character then it should work fine or it will treat them as as two characters or three characters or whatever the number is.

[00:58:36]>> Okay, any further questions?

[00:58:44]Peter muted.

[00:58:46]>> Yes.

[00:58:47]>> I I This is >> um the problem that I found particularly annoying is this.

[00:59:03]>> I don't hear box.

[00:59:06]>> Hold that one.

[00:59:06]>> Sorry about that.

[00:59:07]>> Try now. A problem that I found particularly annoying is the fact that um compound words from chemistry and uh other uh medical terms.

[00:59:25]There are proper hyphenations uh or allowable hyphenations in many places but the preferred ones are between the compounds and it's impossible with the original patch genen to distinguish between them.

[00:59:45]Is this something that can be handled with the uh improved patch gen?

[00:59:56]>> You can you can somehow assign weights to to hyphenation points that you want to prefer.

[01:00:05]So in a sense that should be possible.

[01:00:10]Uh but I think it it's possible also in the old pot gym.

[01:00:15]>> Yes, >> maybe just just the just the range of weights is smaller.

[01:00:22]>> I haven't found a way to distinguish the the waiting of the different uh hyphenation points.

[01:00:35]Uh yes, one cannot by now. But uh as with increased number of levels uh you may have uh patterns that uh uh in luate distinguishes between uh results achieved by levels one to four or levels uh up to 16 uh let's say. And this way you may have in one pattern set uh different uh hyphenation points encoded and that's due to the increased number of levels and similar technique might be now used for multil- language uh patterns uh and uh other obscure ways how to uh encode language information for words in patterns.

[01:01:33]Is that good or do you want me to >> I'll be in touch.

[01:01:42][laughter] >> Thank you.

[01:01:47]>> That's actually a threat.

[01:01:49]>> I'll take that.

[01:01:51]>> And I think No.

[01:01:55]>> Are there further questions?

[01:02:02]Sorry about >> have to be loud voice I'm afraid.

[01:02:05]Charles.

[01:02:07]>> Uh, hi Charles stats. Um, you mentioned that the, uh, UTF Pat gen is has slightly lower performance than the original and um, >> anything now?

[01:02:23]>> I can I can hear clearly. So, I can answer the question.

[01:02:28]>> Okay. So anyway, I mean from a practical matter, I'm guessing that's um totally the performance difference is pretty much irrelevant, but um I am curious is the um difference just an expected result of using a more complex setup or you know having more options that take up more space or is it um some is it a result of the older one having been around longer and been been subject to more optimization.

[01:03:05]>> My opinion is that uh for for instance the dynamic memory ele management may cause a bit of slowdown as opposed to the static sizes which are just really easy to compile and uh optimize in the by the compilers.

[01:03:27]But yeah, uh there might be some super optimizations in the old code as well.

[01:03:40]>> Yep. You're welcome.

[01:03:44]Are there any other questions besides will this microphone work?

[01:03:53]>> Okay, we seem we seem to be good. So let's thank our speakers again. It was really great pleasure.

[01:04:00][applause] >> Thank you.

[01:04:05]>> Nice to see you again, Peter.

[01:04:07]>> Okay, so we'll be resuming at the top of the hour, so you have just a couple of minutes to chat.

[01:04:42]Take that out of your way. Okay.

[01:04:49]So, I'm sort of waiting to the top of the hour >> in case people >> tune in late. They'll get you right from the beginning.

[01:07:12]He recently to resume.

[01:08:55]>> Okay, so we're ready for our next presentation. Our presenter is DTA Verna. He's speaking on quantum boundaries for last line adjustments.

[01:09:07]>> Thank you very much. So yeah, um this is yet another episode in my uh long series about the fine-tuning aspects of paragraph justification. I hope to have the series aired on Netflix at some point, but not right now. Um >> coming this fall.

[01:09:24]>> Yeah, exactly. uh this time I have focused on the fine-tuning aspects of the final line of a paragraph. So I will not insult this audience by explaining what glues and boxes are.

[01:09:37]I'm supposing that it's fine. Uh but let me just quickly restate the the nice trick that the kook plus uses for the final line for adjusting the the length of the final line. So essentially the final line in paragraph is just like any other one. It's a sequence of boxes and glues and everything except that at the end of the final line there is a special glue which has uh no width. It has a zero width but it's infinitely stretchable. Okay. So this is a nice trick because when uh the can plus attempts to justify the final line, it essentially doesn't need to know that the final line is a special case. So there are two different situations. If the final line of a paragraph is naturally over full as uh demonstrated on the first line here in this illustration, then it will be shrunk to the the paragraph width as any other line. But in the case that the the natural width of the line is below the paragraph width, then the can plus essentially tries to adjust it to the full paragraph width as as any other one, except that to do that it needs to stretch it a lot. and all the stretching required essentially is absorbed by the infinitely stretchable glue at the end.

[01:10:51]So the net effect is that the final line actually remains at its natural width.

[01:10:55]Okay, so this is a nice trick that the the algorithm uses. So this is a cool trick but it has some drawbacks.

[01:11:03]uh one uh one fairly well-known problem is that if you are uh in a situation like this one where the final line is at its natural width and the other one is loose, you would prefer the final line to be stretched a little bit in order to be uh visually more coherent with the previous line. In other words, what you would prefer to have is something like this. Okay, this is subtle, but my whole life is devoted to nitpicking stuff like that.

[01:11:31]And yes, I do see my shrink once uh [laughter] okay but as we will uh soon discover this is not the only problem that there is and in fact if you go into really consensious typography in other words if you go see the shrink twice a week instead of only once there are a number of different situations that you would also like to avoid. Uh and oh no the demonstration is not supposed to be happening right now. Uh so probably I need to go to the next slide. Okay.

[01:12:02]Yeah. No, no, no problem.

[01:12:04]>> Um okay. So there are a couple of additional aesthetic considerations that you want to take into account and those things are also well known because uh Bringhurst for instance talked about this uh years ago. So the first uh case that you would like to avoid is what's called runt lines which are final lines which are extremely short. Okay. So this is actually an interesting case because it's it's bad for at least in my opinion three different reasons. The first one is that this final line is just in uh in in absolute terms too short to be actually really satisfactory from a purely visual point of view. The second problem there is is that the final line is constituted of only one word. So you have two consecutive micro interruptions in the reading experience. the eye reaching the the end of the one before last has to stop to go to the next line only to find a very short thing that ends the sentence. This this is also the second reason why this is bad. And on top of that, if the length of the final line is so short that it becomes the same order of magnitude as the paragraph indent, this is even worse. Okay, so this is something that you really want to avoid. And then we have the analogous situation for the for lines which are too long and these are called uh full outlines. So here the situation is not particularly pleasing because this line is actually almost the paragraph width but not exactly. So it gives you the impression that there was an attempt at justifying completely but it failed which is actually not the case. Okay. So here the situation is even more complicated to to handle because you have two different possibilities.

[01:13:44]Well maybe you don't have any possibilities maybe one but but in the the best case you have two possibilities. You could either try to completely justify the line. It's better. But if uh it's also possible you could try to shrink it in order to make a distinction between a failed attempt at justifying and something which is voluntarily too short shorter than the paragraph width. Okay. So these are also uh aspects that we want uh to take into account.

[01:14:13]Okay. So um in my experiments I uh so uh what I uh what I did was uh uh define two thresholds which uh become additional parameters in the justification algorithm like you know we don't have uh so many yet. So two two more is good. uh so uh I'm defining the runt threshold which which is the limit at which you do not want a short line and we have also the full out threshold which is the limit which you want to avoid. So either you need to be below or you need to be completely justified.

[01:14:46]Okay. So in my experiments uh I think that 50 50% of the paragraph width for the run threshold is nice and 5% for the full out one is also nice but these parameters are adjustable. Okay.

[01:14:59]Obviously, in any case, these define two avoid zones uh so to speak, which are the zones in which you absolutely do not want the final line to end. Okay.

[01:15:10]All right. So, uh before explaining how the algorithm works, I need to give you a a crash course on the data structures that I'm using in my implementation of the ko plus because it's really different from the original implementation. It uses a set of object-oriented data structures and so you need to get some acquaintance with those in order to understand how the rest works. So there are two set of data structures one for the problem domain and one for uh the implementation of the algorithm itself. So in terms of problem domain what we have is an abstract class at the top of the hierarchy which represents every kind of breakpoint there is. So these are actual actually actually type setable items and then uh that's obvious a discretionary is a special kind of break point and then an ephonation point is a special kind of discretionary as done in the original uh system then a glue is also a special kind of breakpoint and I also have a special class for uh which is called EOP anyone EOP >> end of paragraph obviously. So this break point is instantiated once in the in the run of the of the algorithm and it tells the algorithm this break point indicates that we have reached the end of the paragraph. Okay. Um a couple of notes about this. [clears throat] Uh as you can see in my implementation of the class penalties are not type setable items. So you you do not insert a penalty somewhere. penalties are actually properties of every single type setable item there is. So th this gives me this is unrelated to the to this this talk but it gives me the ability to fine-tune uh every single penalties for every glue differently everyation pawn differently and stuff like that and when I heard uh the previous talk I think there's something to be done with waiting points which could be done extremely easily with the system but anyway this is not related to this talk what's related to this talk is that as a matter of fact you guessed probably that in order to adjust the final line, we cannot use the trick, the original final infinitely stretchable glue trick anymore. So, we need to get rid of this final glue. And actually, it it's already the case in my system. I do not have a final glue in my uh horizontal lists because uh this is replaced by the special end of paragraph boundary which I have. Okay. So, this is the problem domain data structures. Now, for the algorithmic data structures, so I have something which is called a boundary. A boundary represents a particular break point during the run of the algorithm associated with a specific line. Okay, the the beginning of the line is not represented because I don't need it. But this is in association with a beginning of a line. So a boundary contains a reference to the actual break point at which we change the line. Uh there is the minimum width, the maximum width. Uh any Slavic attendees in the audience? I think there is at least one. that s is not what you think.

[01:18:14][laughter] uh in the case of the kof blast because I have different uh different type setting algorithm in in my system there is a subclass of boundary which is KP boundary which stores additional properties related to the to the kouth plus version of the paragraph justification algorithm and as you can see a boundary for the kouth plus uh stores the fitness class of the line the badness of the line and the local demerits of the line as well. Again a couple of notes. Uh the maximum and minimum width of the line represented here obviously are not the natural ones but they take into account the current tolerance that is uh set by the algorithm. Okay. So it can vary depending on your setting of the to the current tolerance and uh so what I call the tar in this system is actually the acronym for theoretical spacing adjustment ratio. This is the amount of elasticity that you need to use in order to stretch or to shrink the line to the target justification width that you want to use. Okay. So far so good. All right.

[01:19:19]So now it's time to Oh yeah, sorry. No, it's not yet time to see how it works. I have to also uh recall something very important uh theoretical point about the way the KN class works. Remember that the paragraph justification problem is essentially a single pair shortest path question in a graph of possibilities.

[01:19:40]But I insist on this. This is extremely important. We use a graph to represent the possible break points and the path the paths from the beginning to the end of paragraph and not a tree because if you were to use a tree you would completely blow the memory of your system. Okay, pretty quickly because the theoretical complexity of this is exponential. So this has an extremely important consequence. If you're using a graph as opposed to a tree, it means that the nodes or the boundaries in the representation are shared by all uh possible solutions. In other words, what this means is that the way you will continue going down the graph cannot depend on what on how you reached a point uh in the in the current path that you're pursuing. Okay? If you've reached a certain position in the paragraph, the way you reached it cannot have an influence on the remaining of the solution. Otherwise, you cannot share nodes. You need to duplicate the notes.

[01:20:38]And this actually is something which already happens for two different reasons in the original koflass.

[01:20:44]The first one is in the case of non-rectangular paragraphs because if you end up on line seven with different break points the the the the or on line eight for instance the the lines below are not necessarily going to be of the same width and then the solutions are going to be different. Okay. And the second reason for not sharing nodes in the graph is the fitness classes. If you have a single break point with two different lines reaching it of two different fitness classes, then you cannot predict what's going to happen next because you could be inflicted a fitness penalty, a fitness de merit or not. So you need to remember both solutions and so you need to duplicate nodes instead of sharing those notes.

[01:21:29]Okay? And so as we will discover pretty quickly now when it when it comes to adjusting the the length of the final line in a paragraph we need to also duplicate in some situations but fortunately for us not in every case.

[01:21:43]All right. So if you this is what I did at some point I tried to enumerate all the possible situations you get when you want to adjust the final line of a paragraph. And if you can imagine there is no less than 32 individual possible situations. Okay, they're all represented on this figure. What you have here, if if you take this one for instance, this represents one possible line with its minimum width given the the current tolerance, its maximum width given the current tolerance and then the natural width of the line is somewhere in the middle. Okay, so if you want to enumerate every possible case, these are all the cases in question. There are 32 of them. Fortunately for us, the treatment is not necessarily different in every situation and sometimes you can even treat different cases the same way with a different outcome but the algorithm will remain the same. So now we are going to uh see uh what we need to do on these uh specific cases.

[01:22:45]So first uh there is first a whole a whole set of cases for which you do not need any new kind of data structure to represent the line breaks. We you can just reuse the original ones. So these are what I call the deterministic cases.

[01:23:01]So for instance in those situations as you can see every line in those cases is too short or just the required amount to to be justified to at least the runt threshold. Okay. So what I'm doing here is just reuse the uh the classical boundaries that I already have except that I'm going to pretend that my justification target width is not the paragraph width but the runt threshold.

[01:23:27]So I'm computing the tar accordingly and I get different situations in cases one and two the line is too short. So the tar is going to get a badness exceeding the tolerance and the line is going to be considered as under full and it will just be eliminated as a regular line by the by the algorithm. Okay. In case three there is only one possibility the line can be extended to exactly the run threshold and so it will be and in case nine there is no choice. The line is not elastic and it happens that this basically never happens. Okay. But from a theoretical point of view, you absolutely need to handle every possible case. And so in that situation, the line's already at the only width uh available and it happens to be a correct one. But that is really miraculous.

[01:24:14]Okay.

[01:24:15]>> Okay. So case deterministic cases number one. Number two, it's sort of the the the the analogous situation. So in those two situations, what you can do is adjust the line to be the full out width. Okay. So I'm again computing the tar accordingly and in those two situations the outcome is the same. The line is adjustable and we can use it.

[01:24:38]Okay. So in in in in case 21 the line is already a fine and in case 22 the line is shrunk to the full out width. Okay.

[01:24:48]Interesting uh cases number three.

[01:24:53]I'm still using the same data structure but this time the outcome will be different. In all those situations, there is only one possibility which would be to fully justify the final line. So I'm going to pretend to do that. But the outcome will be different.

[01:25:09]In cases 27 and 28, it works.

[01:25:13]This is the only possible solution. In cases 29, there is nothing to do. In cases 30, it also work, but not by stretching, by shrinking. And in case 31 and 32, the line will be considered over full as uh before as a regular line in the algorithm. And so it will be eliminated by the algorithm as before as a regular line. Okay. So far so good. Am I making any sense? Yeah. Okay. Fine.

[01:25:42]And now this is the time when we enter the quantum world. Of course, this is just a marketing term to catch the attention. But there is still a a kind of a nice uh analogy with quantum mechanics. I'm going to explain why.

[01:25:56]There are a number of interesting situations in which you end up having a lot of possibilities.

[01:26:02]But because those possibilities depend on the one before last line, you cannot yet make a decision at least if you want to continue sharing the nodes in the graph. Okay. And so the idea that I have here is is in fact quite natural once you come to think about it. At that point in the processing I do not know what to do. I cannot make a final decision.

[01:26:25]So let's not make a final decision.

[01:26:27]Let's carry the uncertainty as long as possible. And this is when we enter the world of quantum mechanics because we have boundaries which are in sort of an indeter indetermined state. they are in multiple state at the same time and the final state is only going to be decided once a final and actual paragraph solution is observed by the observer. Okay, so this is the analogy with quantum mechanics. So in those situations you have basically two possibilities. You can either shrank shrink the line to the full out threshold or you can try to attempt full paragraphs justification. So I do not want to make any decisions. So I'm keeping both possibilities and hence the the existence of this new class which will store the full out uh spacing adjustment ratio. So the amount of shrinking or stretching that I need to to to get the full out width and the justification spacing adjustment ratio which is the one required to reach that point. Okay. So I'm just keeping those two values around. In that situation, the tsar, the original one, because remember that this class gets a tar. So this one is a subclass of this one. So it also gets the the normal tar. And so the the regular tar is going to be initialized to a minimum of those two values in order to remain as close as possible to the natural width of the of the boundary cases. Uh um a couple of other cases using the same class. This time as you can see no solution is possible but I want to express the fact that these two that these two cases cannot be either shrunk or stretched. So I'm keeping those two values again. And in some sense it means that uh this uh a line like this one is actually going to be both over full and under full. Okay.

[01:28:24]Quantic quantum quantumly speaking.

[01:28:27]Okay. It's going to be over full with respect to the runt threshold and under full with respect to the paragraph width. Okay. Are we enjoying ourselves so far? Okay. So, uh for a reason that I'm not going to explain the is going to be initialized to the justification adjustment ratio. I don't have time to explain why but if you're interested I can give you the explanation uh of a coffee.

[01:28:54]uh two additional cases where another kind of quantum boundary is required. In those situations, the situation is uh the the cases are more complicated because for all these lines you have a full range of possibilities plus the possibility to be fully justified. Okay, but again remember that we want to avoid this zone. So there is a whole range of possibilities. Then we jump quantumly speaking to the paragraph width. Okay.

[01:29:26]So again I'm going to store that information. The minimum spacing adjustment ratio is the one which is going somewhere here. The minimum that you can do. The maximum is the maximum that you can do within uh this uh interval and then you also have the JS to represent the adjustment ratio needed to reach the paragraph width. Okay.

[01:29:49]final case and then we can go back to uh our good old deterministic life. Uh this case these cases use the same uh data structure but the difference is that those lines cannot be fully justified.

[01:30:05]So the situation is exactly the same except for the fact that the JSR is going to be set to uh nil indicating that it is not possible to to reach the full paragraph width. Okay, how am I doing uh with time? Extremely bad as usual. Okay, so these are uh these are all the cases. Now uh in terms of sharing, let's get back to that particular problem. Uh as as you understood um in the deterministic cases, we don't have any problem because we're just reusing the old data structure. In the quantum cases, there is a problem obviously because at some point we have to make a decision. So I'm not going to enter into too much detail here but let let me just mention that at some point we need to duplicate the notes and so we need to get rid of sharing. Okay but this is going to be done at the final stage at the a as late as possible in the process. Okay, so when we're considering an actual path, we need to fixate the state of the quantum boundaries and in order to not disrupt the sharing of the nodes, the bound the quantum boundaries are going to be duplicated and the actual SAR is going to be fixated once and for all depending on the state the state the appearance of the previous uh line and and that's about it. Okay.

[01:31:26]uh and obviously the adjustment consists in getting as close as possible to the tar of the one before last line given what's permitted by the uh quantum uh boundary uh uh restrictions.

[01:31:42]Is that clear more or less? Okay. Uh yes. One um one thing extremely important that question could be debated again over coffee not right now but there is also an implication with this algorithm which is that when uh when you stretch the final line the question is whether to take into account the badness of the final line. The koflass takes the badness of the final line into account only when it shrinks it but that's because it never stretches it. Okay. So in some sense the question could be debated because stretching the final line you want to do that in order to get something more beautiful more coherent with the previous line. So what sense can it have to stretch the final line in order to be more beautiful but then increase the badness because of stretching which which is a way of saying no no this this is going to be bad and better. Okay. I still think that stretching is uh taking into account the stretching badness is important and this is what I do but this question could be uh debated just not right now. Uh oh yeah so now I need to get back to the sorry yeah I need to get back to the demo.

[01:32:59]>> Okay so this is a demonstration of the algorithm uh in real time so that you can see what's going on. So at the top here, I'm going to trigger the visual clues for the two thresholds which you can see here. Okay.

[01:33:13]Um so these are the two the run and the flatout thresholds here. These are the sliders which allow you to adjust in real time the values for these two thresholds. And we're going to look at a different situations.

[01:33:24]So I'm going to increase the fullout threshold progressively. and you will see the final line stretch shrinking in order to fit the constraints to a point where it's not possible to fit it anymore and so another solution has to be chosen by the algorithm and then again it's going to happen and this time the only possibility was to find another solution which happens to be completely justified. Now let's do the same thing for the run threshold. So I'm increasing the run threshold. So this line has some elasticity. So we're able to stretch it a little bit. At some point we can't anymore because that would exceed the tolerance. So we need to find another solution and uh etc etc. My thresholds uh stop at 50%. So uh this one is the final one. And here is yet another situation. So this is interesting because there's no elasticity on this line. So if the the threshold increases I need to find another solution.

[01:34:21]So now we're going to directly jump to yet another solution because the the final line even though it had some elasticity did not fit the the constraints of the algorithm and so we had to resort to another uh completely different solution.

[01:34:37]All right. So this um yes okay so that's how it works. Now what I did was uh study the impact of this algorithm on the general outcome of the paragraph justification. So I did a lot of experiments uh I did more than a thousand experiments on paragraphs coming from the mobidic novel and every time I run the original koflass algorithm and my uh version with the final line adjustment to see the the behavior and the impact on the result.

[01:35:07][clears throat] Um so there is a number of um there is a number of statistic uh result which are I think uh worth uh studying. So first of all failures in in the original case the canoe plus completely fails within the uh within the tolerance that you give it uh in 33 case which is 2.5%.

[01:35:29]Okay so that is the cases where the original one can cannot do anything. So you have to reformulate or increase the tolerance or do something sloppy or whatever. uh if you take into account the new parameters from the new constraints with the final line adjustment I got only four additional failures which is not a lot. Okay. And again then I what you need to do is either adjust the thresholds to to to let the algorithm perform correctly or maybe increase the tolerance blah blah blah. Um here is an example. This in this in this case there were actually only two solutions all in all. Okay. So the first one is on the left, the second one is on the right. And as you can see, these are the only two possibilities for this paragraph. And in both cases, the final line is the same. And it doesn't have any elasticity. Um the the gray area represents the the threshold. Okay. So normally you would like the line to be at least that long, but you cannot possibly do it. So it fails in the first case and it fails also in the second one. And and we've run out of possibilities. So there is nothing that you can do here.

[01:36:38]Um another interesting result is that in the vast majority of the cases the solutions are the same in both versions of the algorithm. But be uh and that is 86%. But beware when I say identical I mean the break points are the same but then even though the break points are the same the final line is going to look different because it's uh its width is going to be adjusted. Here are a number of examples. This one is is a case where the original one was already satisfactory because it fell into the the the the bounds the the two thresholds. Okay. So this one is at its natural width. But as it happens the the one pre the the previous one is actually loose. So I don't know if you can see it. It's a bit subtle but uh uh in my version of the algorithm the line this final line has been stretched uh a little bit because this one is stretched the previous one is stretched. Okay. So this is more coherent with the previous one. Uh there are also a number of cases where the original line is not satisfactory but yet it is adjustable.

[01:37:47]Interestingly enough, I I discovered only one case of an original runt final line which could be adjusted. This is probably not surprising because rent line are very short. So the likelihood to have uh enough elasticity to do something is is really low and sometimes the the run lines are not elastic at all. So there is nothing you can do.

[01:38:10]Okay, I'm not going to show you this example because you wouldn't see anything in this particular situation which again was the only one. The line was too short by 66 points and so if I were to show you the the the original one and the adjusted one, you wouldn't see anything on the screen. Okay? So, it's not worth displaying anything.

[01:38:33]Um case uh other cases where the line is originally full out. So here is an example. As you can maybe see, I'm not sure about the the back of the room, but as you can see, this one is slightly full out. So it's slightly too long. So what you need to do normally is to shrink it to at least the full out width. But if you look closely, you will see here that the line is in fact shrunk more than the full out width. Why is that? Again, because the line above is tight. It's not decent. Okay? So this line has some shrinking. So this line has been adapted to at least the full out width, but it's been shrunk even more to get closer to the one before.

[01:39:16]Okay.

[01:39:18]All right. Um and finally adjustable overalls going not uh sorry going to be shrunk but to the full uh to to the paragraph width. So this is 5% on the of the cases only. So that means that the original final line is naturally over full. And so again there is no point in showing you an example here because when a line is is originally over full what you need to do is compress it to the paragraph width. But this is exactly what the kof plus uh the original version does. So if I were to show you both versions you would get the same picture. Okay? So there's no point in showing anything. All right. And then more interesting situations.

[01:40:02]I'm I think I'm out of time, but I have a deal to propose. Give me just five more minutes or I will not organize tug next year in [laughter] [applause] Does that sound fair? Yeah, you're okay.

[01:40:18]I by the way I took the precaution of asking the audience and not the session chair. [laughter] Well, okay.

[01:40:26]uh interesting situations where the the new algorithm has to come up with a different solution than the original kof plus and again there are also different sub cases within uh those cases. So first one when it's uh uh when you need to choose a different solution because it's necessary to do so. So again in this situation the final line doesn't have any elasticity so there is nothing you can do. you absolutely need to find another possibility and so the possibility that the the algorithms the algorithm has come up with is to stick two more words on the final line so that it is sufficiently wide. Okay, so these amount to 75% of the cases where the situation is different. Okay, even more interesting when it chooses a different solution but it didn't really have to.

[01:41:21]Okay. And that is because of the stretching badness take be taken uh into account. So let me explain what happens here. This line as you can see is perfectly okay for the original algorithm but also for the new version because it falls perfectly uh well within the bounds of the run and the fallout threshold. Okay. But what happens is the following. The the line before is loose. So what we want to do is actually stretch this line to the same amount as the previous one. When I stretch the line in question, the badness of this line increases because the stretching is taken into account to compute the badness. And remember if you know that that the fitness classes in the KN plus are actually quite coarse. So what happens here is that the badness of the line increases but the fitness classes do not change. So it means that the global the total demerits of this paragraph increase as well to the point that it becomes greater than another solution.

[01:42:32]Is that clear? Yeah. Okay. So this is a case where the algorithm because it has taken the badness into account chooses another solution even though it was not a necessity.

[01:42:46]Okay. Now I have a confession to make.

[01:42:49]If I'm being completely honest I think that this one is better than this one.

[01:42:54]But again over coffee if you're interested but this is not the fault of the algorithm. The reason is that the the fitness classes in the gnof class are too coarse and that there is also another uh aspect which is not taken into account at all in the algorithm which is that for instance it's not very nice to have a word followed by a punctuation mark as the beginning of a of a line. So admittedly this is something that you would need to take into account as well. Okay that was over coffee actually.

[01:43:22]Fine. Uh to conclude this a couple of additional figures very quickly. I've uh computed the pruning ratio of this whole thing which means that given the additional constraints of my algorithm there are a number of solutions which were possible before which become completely impossible now the pruning ratio is actually of only 7%.

[01:43:44]I was about to say unfortunately for us this is not going to have any impact on the performance because you you you can only figure out that the solution is impossible only at the very end of the processing because you need to to to to um to pass through the whole paragraph before making a decision. Okay. So this this might be a an interesting theoretical uh discovery but it's not going to have any kind of impact on the algorithm itself. uh the average lookup death is also to 2.11. What does this mean? It means that when my algorithm choose an a different solution than the original one, the solution in question is in average the third choice of the original can.

[01:44:29]Okay, this is what I call the lookup death. And so it means that the solution that I have to find is not really far away from the best one in the eyes of the koflass. So, I mean, congrats, Don.

[01:44:41]You know, it means that it already does a pretty decent job at doing all that stuff without all the without entering the quantum world, right? Okay. And finally, this may be of more interest to professional publishers. I also uh measured the average length of the paragraphs in both uh cases and I found that in 20 to 30% of the cases when again the the solutions are different the new solution that is uh selected by my new version of the algorithm is us is shorter by one line never more than one line but one line shorter okay so this is probably something which is well it's not surprising because again long lines can be adjusted and most of the time line too short has to be the the for instance the final word has to be stuck above or something like that. So this is probably not very surprising but it's also an incentive that you can both be better in terms of consensious type setting and also maybe hopefully save some paper. Okay. Uh and uh that's it.

[01:45:46]So to conclude I guess I'm going to organize in Paris after all.

[01:45:52]Um, so as we saw, fine-tuning the final line of the par of a paragraph is not maybe not as trivial as it may seem at the first glance. Uh, essentially because you need to get rid to some you need to depart to some extent from the original algorithm and you need to get rid of the the trick the the final special blue trick. uh we've seen that it's not only a matter of adjusting the spacing of the line but sometimes out of necessity or by choice you need to um to come up with a completely different paragraph breaking solution.

[01:46:27]Um the algorithm that I uh that I have proposed is able to do both. is able to to compute new solutions and at the same time adjust the width of the final line to be closer within the authorized range uh to the final line and at the same time it's it's able to avoid the rent threshold and the fallout threshold. Uh the core idea which again is uh not so complicated is to carry the uncertainty until the very last minute and that's the way you can get around the problem.

[01:46:59]And finally experimentation has shown that in 86% uh of the cases the solution the the breaking solution is actually the same which is which is actually quite comforting. Uh rent lines as we've seen are also but this is not a surprise rarely adjustable and also interestingly enough in 20 25% of the cases alternative solutions chosen by the new algorithms are due to the fact that the stretching badness is taken into account. Okay. And that's all I have.

[01:47:34]Thank you.

[01:47:37][applause] Well, thank you so much. Um, uh, so in by the power vested in me as the person [laughter] running this session, it is break time and you are allowed to leave if you want, but you are not allowed to leave as your penalty for going over because you must stay here for questions.

[01:48:02]>> I will happily comply with that order.

[01:48:04]>> Okay, that's the power vested in me. So really, if you want to sneak out, that's okay. But if >> but I'm I'm taking the names.

[01:48:12]>> Yeah. So Oh, where's our uh >> I can give you this one.

[01:48:18]>> Oh, no. I needed to answer. So >> Oh, mine. Huh? Okay. Okay. So >> yeah, >> I have a question from the chat. Uh people ask whether it plays well with microtype extensions.

[01:48:34]Will it work with microtyping extensions?

[01:48:36]>> Yes.

[01:48:36]>> Uh I don't see why it wouldn't.

[01:48:39]>> Uh my my ultimate goal is to incorporate as many fine-tuning aspects including auto typography and micro typography in the core of the algorithm. So yes, the this algorithm is just a matter of avoiding the the thresholds. Currently I do not have the much micro many microtypographic extensions in the core.

[01:49:00]But if if for instance you have a run line which you cannot add which you cannot stretch by the usual means but if you do some tracking or if you slightly modify uh the kerning and it's sufficient to get to the run threshold I'm I'm extremely happy with that obviously.

[01:49:19]>> Thank you >> Eric.

[01:49:27]>> So >> or expansion. So which uh how how did did you uh override the the original uh model like >> uh I I've re-implemented everything. So I have a completely I have a complete uh new implementation of the original algorithm and lots of different algorithms and and lots of variations which I can activate or deactivate in my platform. But this is this is not done in a production engine. This is done in my experimentation platform which is a complete rewrite of everything.

[01:49:59]>> And and will you also be sharing that?

[01:50:02]>> It's already on GitHub. Okay. So you you can download it. It's uh one drawback for most people is that it's written in lisp. So you have to get you see pavit >> I I I managed to stuck lisp in the talk after all.

[01:50:16]>> Uh but but it's it's essentially you can use it uh from the from from the list prep but it's also a gooey. So if you even if you do not know anything about the the the underlying programming language, you can play with the sliders and everything.

[01:50:30]Eric and everybody has written several papers about his platform previous experiments and they're all in talk about sorry people online [laughter] >> you could just say that again >> yeah uh uh I have several papers about the platform itself at the at the tug conference also at uh at the European lisp symposium conference and everything is uh accessible online. So if you go to the to GitHub, you will find the platform and you will find probably links to the papers that I've written about the platform itself as well.

[01:51:06]>> No, sorry, I meant clickable papers. [laughter] Uh, quick question.

[01:51:15]>> Yeah.

[01:51:17]>> What about a full what you call a full outline? full outline >> that would be justified and happened to be the last line of the page.

[01:51:27]>> Uh as the reader I would expect that the paragraph would continue on the following page >> but then uh it would be over. [laughter] >> Okay. Uh I see what you mean. I'm I'm not sure I can get an example here >> because being nonjustified is a sign that >> Yeah. Um So okay the question you're asking I have a extremely convenient answer for me convenient for me I mean the question you're asking is beyond the scope of this work [laughter] the the reason is that what you are talking about is uh in uh two paragraphs interacting with each other. Okay. And the the original algorithm does not really do that. You could probably do something related by uh introducing a new kind of interparagraph penalty or something. Uh but but this is still even though it's out of the scope of my work, this is still a very interesting question and there are a lot of different aspects that you would like to take into account. For instance, uh if two paragraphs um uh without any skip in the vertical skip in the middle are so next to each other [snorts] and the final line of uh of the previous one is tight but the beginning of the next one is very loose.

[01:52:50]This is not really visually pleasing either. And that is the kind of stuff that you would like to be able to adjust as well. But that is not done right now though.

[01:53:03]taken into account.

[01:53:04]>> Yes.

[01:53:08]>> The first part of the paragraph is completely irrelevant to the second part of the next page.

[01:53:15]>> Yeah.

[01:53:16]>> So for example, the tightness of the last line and the first line shouldn't be part of the solution, but it is of course because you first do the other one. Same here the first paragraph and then you have a page >> surprise.

[01:53:36]>> I would uh along with these lines I would also mentioned that there is a at least in France I I don't know in other countries but at least in France there is a debate among uh professional uh compositors. Some have a tendency to think that the final line of a paragraph should never be completely justified.

[01:53:56]um and some others I um um I tend to agree with the other camp which says that it's actually not true anymore because full justification was a problem or is a problem if you have a power indent of zero but since a paragraph indentation of nonzero gives you the clue that a new paragraph is beginning having the last one ending with a blank is redundant.

[01:54:24]Okay, maybe I'll use the power vested in me. You >> can I go get some coffee now? [laughter] >> Not till everyone else leaves. They deserve it.

[01:54:32]>> Okay, fine. I'm the last one on the ship. No problem.

[01:54:35]>> You're the captain. Get out of the ship.

[01:54:37]Okay.

[01:54:38]>> Thank you very much again.

[01:54:39]>> Right. So, but we will start at 11 o'clock. You know, people are logging in from outside and want to hear the next talk. [snorts] So, uh um out you go. Come back in. Thank you.

[01:54:51][applause] have uh an extra announcement concerning lunchtime that always gets that always gets everyone's attention.

[02:09:04]>> Well, we could do it at lunch just before lunch.

[02:09:08]>> Yeah. Well, I'll say it anyway. There's going to be an analog playground set up >> during the lunchtime to play with fountain pens, inks, and papers and that sort of things. The the person to see is right over here.

[02:09:23]>> So sitting on the right now a little further. That's your chair.

[02:09:36]I learned something.

[02:09:48]>> Okay, so we're now ready for the next talk with everybody refreshed with coffee and little snacks. So the next talk is entitled how to make a good first impression which certainly uh bodess well for a conference like this.

[02:10:07]The speakers thank you Michael. Um I won't use this big microphone. I'm used to project my voice. I mean for me it's a small audience. I usually teach to uh classes that are two or three times this uh this size. Um yeah, so um I'm it's actually my second visit in a TUG conference after the 2024 edition in Prague and I'm here representing Satan.

[02:10:42]Life's wonderful. Um not probably not because I'm the most deserving person, but uh I was the closest one uh since I live in this country. uh but in Quebec City which is a mere 4,000 kilometers to the east.

[02:10:59]Um yeah so um it seems that there was uh some interest in uh in this talk about how to make a good first impression when you submit a package to Satan. So essentially it would try to be sort of a guide of how to be accepted on your first upload and as I will mention this doesn't happen very very uh often.

[02:11:25]Um so I'm representing Sitan uh they don't they don't seem to have a template for slides so I just used my just used my universities. Um so uh I'm also a professor at University Laval.

[02:11:42]Okay. So uh let's go. Uh perhaps uh before I start I could just say a few words about Sitan. The the the the objective of this conference of this presentation is really not to give a lot of details about Satan. Uh here's uh just a quick outlook at the Satan team uh using these nice little pictures that were uh made by our uh web master uh gird. Uh you can see that although the site is pretty important, it plays an important role in the community, it's a fairly large site, the infrastructure is remarkably complicated. Uh but very very few people are there to make it work. Um um in well the N in Satan means network. Uh originally there were three nodes of Sitan. One in the US, one in the UK and one in Germany.

[02:12:42]Guess which one is remaining now.

[02:12:44][laughter] Okay. Um there's still a place here. Um normally it should be Sachi who who should uh come here. Uh Sachi volunteered last year. Uh we didn't have the opportunity really to start having him uh contribute to the team. Uh it should uh well we should be able to kick it off now. And I'm sure that uh once a G has made a drawing, it will represent your hair better than mine [laughter] because it it's either the beard is is too dark or there is none since the the the drawings are in black and white and you can tell that the color is not exactly the right one. [snorts] Um so this is the team. Uh this is just to give you an idea of the volume of uploads that we're processing uh per month. Uh this drawing actually you can find on the seat sitan homepage just that I redid it. Um so on average uh we treat well you see around 250 to 300 uploads per month. Okay so that's about 10 per day. Uh of those uploads these are the new uploads. So we have between well it's difficult to tell on the scale here. Uh if I just want to be in line with my abstract, I would say about 10 new uploads per month, which would give about one every two days.

[02:14:17]Okay, this varies a little bit. Okay, but this is just to give you an idea of the volume of uploads that we have uh to process.

[02:14:28]Okay, so what's the problem that we're trying to tackle?

[02:14:34]It's the fact that very very few packages uh new packages are accepted on the first upload to sitan and almost none by new authors. So if you compound the two things sending a new upload and being it being your first upload well there are close to 100% chance that it will be rejected on the on the first try. So that's something that we'd like to tackle that that this is what uh we'll uh I I will be discussing with you. So the solution could be read the documentation. It's all documented how you should prepare and upload. Okay. So RTFM.

[02:15:21]So there's a page on the Sithan website how can I uh how can I upload a package?

[02:15:27]This one is fairly is fairly short. And then there's an addendum. [laughter] This one covers pretty much all the the the the error cases that that that we've seen over the years. And I guess it has been the the problems have been stacked.

[02:15:46]Uh so in the end um the whole documentation is about 5,000 words.

[02:15:52]That's about 20 pages. Okay. So it's very long. Uh, and I guess that not that many people read the whole documentation and I wouldn't blame them.

[02:16:05]So, uh, what should we do?

[02:16:10]when I was preparing my talk uh well I knew what I had to talk about because well I had already written the the abstract but um I didn't know exactly how to approach the the the problem or the the talk so I was sort of goofing around pushing it uh uh ahead and eventually I ended up on a stack exchange uh thread uh linked by Max actually [laughter] thread in which there was a link to this paper which is actually just a one pager by none other than Leslie Lamport.

[02:16:57]So it was very very fitting to uh present it here. I don't know how many people here knew about this thing. So uh the the paper basically everything is in the title. It says state the problem before describing the solution. And Leslie was basically saying well he he he says computer papers in computer science you can extend it to any other discipline would be the same. says, well, quite often in scientific papers, we have a a statement of the problem, one solution that is proposed, and then a proof that the solution is indeed a solution to the problem using metrics that we choose such that of course your solution will appear as being the the best one.

[02:17:43]Rings a bell to anyone.

[02:17:45]Uh so basically uh Leslie was saying well do this instead. We should uh instead state the problem then state the correctness conditions. So what would be considered as a correct solution then propose a solution and prove that you are indeed meeting the requirements.

[02:18:07]Um so this is what I will be trying to do but well keep in mind that obviously I use I was using this approach but I trying to pretend that I'm using this one. Okay but let's do it just for the sake [laughter] of the presentation. So what would be the correctness conditions to have a package uploaded successfully on Sitha? Basically, in other words, what are we looking for in a package?

[02:18:39]Okay. And then we'll see what are our requirements to meet these conditions.

[02:18:45]So, what do we want for a package to be accepted on SATA? And bear with me for some of the icons. It's not always easy to find an icon to represent an idea.

[02:18:55]So, first we would like the package to be discoverable on Satan or in the distributions.

[02:19:05]Uh just last week, Ger, our web master, uh informed us that we reached the threshold of 7,000 packages on Satan.

[02:19:16]Okay, so good luck being found inside this whole forest if well the aim of your package is not clear. Okay, so the package should be discoverable. We want to have clear copyright and licensing conditions usually because it is free software. This one is sort of optional, but you know um usually if you write a latte package, you want to be included in distributions and especially in techlive and uploading to sitan is the path to get in inside techlive and to be in tech you have to be free software. Am I correct car? Good. So probably that you want to have to be free software. We want to avoid name clashes with other packages. I'll come back to that uh in greater detail. Uh we want the package to be portable because well the distributions are offered on many different platforms and well there are humans involved in this whole thing. So packages are not processed automatically. They're processed by humans, of course, using some tools, but in the end, they're processed by humans. And we would like it to be we would like the package to be as easy to process by the humans involved.

[02:20:46]So, as easy as possible. Okay.

[02:20:50]So, now that we have these correct correctness conditions, let's see what are the requirements. We have six conditions. I have seven requirements to show you. and uh it's just that some cover two uh conditions and whatever.

[02:21:06]Okay, so solution redux. So let's try to build another solution that is not read the documentation.

[02:21:17]First point, the package name, it's the most common source of rejection is just the name of the package. Okay. So we ask you to choose a descriptive and non-ryptic well obviously what is cryptic and what is not is open to debate. Uh but so try to have a non cryptic name for your package.

[02:21:42]Um of course this is pretty broad. Okay.

[02:21:46]What what is considered as descriptive and non-ryptic? something you can judge as being very very descriptive but only people in your field will understand what it is about. I'll give an a few examples in a moment. So for example uh very very short package names well sometimes when we design the thing on our own computer we want a very short name because it's it's it's faster to type and so on and so forth. Well, very very short names tend to always be re rejected nowadays, especially if there are no vowels in the name. Okay.

[02:22:21]Um, don't put a text suffix to uh your package. Maybe you developed it as thinking, oh, I'm doing this type of thing for for for tech, so I'll add a text suffix. They tend to be reserved for new engines. So, unless you're an engine, don't put tech inside the name of your package. Okay.

[02:22:43]Aim at the universal. Here what I mean is that try to find a name that everybody will understand. So not no don't be too uh say country or culture culture specific. Okay. So something could mean one thing in your culture and mean something completely different in another culture and it it just brings confusion.

[02:23:06]Follow the path when there is one. I mean that for some packages there's all already a sort of a tradition that has been laid out for example beamer themes they're all called beamer theme something so just follow the path okay uh tix packages usually have tix in the name um thesis templates usually have thesis in the name so on and so forth okay and um you cannot say well yeah but there's this package already in tech life that it's not admissible okay it's just that some people have been here much longer than you and me okay I mean there are directories in sitan by the name by the first name of a person okay so you have to have been there for a very very long time to have a directory at for at your first name in uh the the catalog.

[02:24:10]Okay. So, uh let's see a few examples.

[02:24:13]There there are fairly recent examples.

[02:24:16]Uh we received this package Q draw was well not a bad name but what does the Q stand for? Okay. Is it quick? Uh U u there used to be I think a drawing program called Q draw. Uh we really asked the author what what what does the Q stand for? I don't really even remember the the answer but we proposed some other solutions and he came up with something much more poetic than than we proposed art draw. This is pretty cool.

[02:24:50]I think it's a very nice one. [laughter] Um or style. Um to me as a mini as I'm an Emacs user uh org stand would stand for org mode.

[02:25:04]>> Well no the author has had no idea what was orc mode. Okay. So it's it's it was there for organization.

[02:25:12]Okay. Uh so we we re renamed the package as brand styler in the end which was basically a package to take the identity of your organization and make templates for for it. So the R at the end uh was is interesting in this case. AP reacts.

[02:25:35]Well, for someone in the field of atomics physics, AP is a very common acronym.

[02:25:43]But if you're not an atomic physicist, um well, what does AP stands for?

[02:25:50]>> Advanced placement [laughter] >> could be anything. Okay.

[02:25:56]So uh it became atomic reactions which I think you will all agree that it's it's a little bit clearer. Uh pong tech. So this is one where uh there's a tech in the in the name. Okay. And pong >> if you're old enough you think of the old game. Okay. The first computer game.

[02:26:14]>> Uh so it was renamed Majong Tiles. I think it was Car's suggestion.

[02:26:19]>> I'm not sure about that one.

[02:26:20]>> Yeah. Yep.

[02:26:23]>> [laughter] >> Uh, karam was an is an interesting one.

[02:26:29]Karam is a is a type of billiard's game common in France as if >> y uh so it was actually a very good name.

[02:26:41]>> The problem with this one is that and called discovered couple of days later was is the fact that there's already a karam.sty sty file somewhere in techlife. Okay, so name clash for for this one.

[02:26:59]>> So we asked the author to rename the package and actually it's actually clearer this way. Karam billiard.

[02:27:05]>> Yeah, it's just a better name overall.

[02:27:08]And then there's this one that uh occurred recently. XO.

[02:27:14][laughter] >> It's actually a package to to draw tic-tac-toe uh boards. Okay. And that's something I didn't know, but tic-tac-toe is actually the name of the game in the US and Canada, but in other places of the world, it's called XNO or >> knots and crosses [laughter] in the UK and stuff like that. Okay. Um, I put ellipses here because u we still haven't reached a conclusion. the the author did not get back to us uh so far with with a chosen name >> Dutch name perhaps >> here I should just mention a funny story uh really as I was laying out the plan of the my presentation and I had just prepared basically this slide we received an upload the package was named CW [laughter] so two letters no vowels >> [laughter] [snorts] >> So, uh, if I just go back a second, uh, it was certainly not descriptive, certainly cryptic, um, way too short.

[02:28:27]Okay, for the text suffix and everything. Well, it's just that for this one, the upload was by Jim Epheron.

[02:28:36][laughter] No.

[02:28:40]So I was slightly hesitant to tell him well [laughter] you know and it was actually a package to do Morse code. There's already a a Morse package on Satan very very old package and you just wrote uh another simpler version and CW would stand for continuous wave which would be another name for Morse code.

[02:29:08]uh he ended up renaming it as Morse simple because his package is actually simpler than than Morse.

[02:29:17]I thought of proposing walrus >> because Morse is actually the name of a walrus in French. So if you go from English to French and back to English, you get warrus.

[02:29:33]That fails the aim at the universal.

[02:29:38]>> [snorts] >> Okay. So that's for the name. Uh another thing that we ask so that well if you want people to get interested in your package first have a good name is a is a is is a good thing and then easy access to the documentation. I don't know if you're like me first thing you look at.

[02:29:56]Let's have a look at the documentation.

[02:29:58]Okay. So build good documentation for your package also. But whether it is good or not, we ask you to provide some documentation in PDF format so that we can link to it uh from the sitan site.

[02:30:13]Okay. So if the package has documentation, we put a link to it. We do not compile anything. So that's why you have to provide the documentation already compiled in PDF format. We won't do it ourselves. Okay? So that's simple enough. Just provide the documentation.

[02:30:31]That's the only pre-ompiled thing that we ask you to provide is the documentation.

[02:30:39]Then a readme file.

[02:30:43]Well, it probably used to be more overlooked in the pre-Gitub age where readme files were not as common as they are nowadays. So today they tend to be there. Uh so read me in any sort of format you want. So read in markdown in or just pure text doesn't matter. So what we ask of to to be present in the readme file again this is information that will be put on portal short description of the package.

[02:31:14]Who are you? How to contact you and what what is the license of the package.

[02:31:18]That's basically what we ask for in the readme file. Some authors tend to give a a lot a lot of documentation in a readme file doesn't doesn't have to be there.

[02:31:30]Um but um this is the information that that we ask you to provide.

[02:31:36]Uh be careful the readme file the short description should not ideally in my opinion be a replacement for proper documentation. So I tend not to like personally a package that has a long read file and no no proper documentation. Okay. I think the idea of the documentation is um documentation in in PDF shows greater care in my opinion.

[02:32:02]Okay. So so far so good with these three um components. So, a good name, a readme file, and proper documentation. We've covered a good chunk of the discover discoverability aspect of your package and also the clear copyright and licensing conditions.

[02:32:27]Next, if you want your package to be free software, you have to provide the sources of your package. And this is another place where uh some errors uh occur.

[02:32:40]So what we ask for in the sources is no generated files. So everything that could be generated from the sources themselves should not be included in the package with one exception the documentation. So we do ask the documentation but nothing else. Okay. So for example, if you're doing your if you're building your package in using literate programming and the sty file is extracted from a DTX, well just provided the provide us the DTX. We don't want the STY. Okay, so no generated files. Uh ensure coherent licensing. This is something that we see quite often. One file says uh LP LPPPL 1.3. Another file says one LPPPL 1.3C.

[02:33:33]Petra doesn't like this. [laughter] Okay. So try to have the same licensing conditions everywhere. And here I'm not talking about you could have two different licenses for different files in in a package. For example, the package file itself is LPPL and the documentation is uh C create C creative comments. That's fine. Okay. you have you may have two licenses. Uh but if you're using one license, well, try to use the same license everywhere.

[02:34:06]No git repository, please. There's a double anton there.

[02:34:13]First thing is not everything that is part of your git repository has to be in the package. For example, you have a make file to build the package.

[02:34:27]What good is it for end users to have your make file? It's probably part of your git repository.

[02:34:36]There's no real point including it in a package. So it's not because a file is part of your repository that it has to be in the package.

[02:34:47]So that's one thing. The other thing and it goes with the fourth item. We don't need the git repository inside the zip file because you'd be surprised but it happens quite often because the git repository is a hidden directory in the directory of uh the package and people will just zip the whole thing and send us the whole git repository in the zip file. Okay. So, be aware be be aware of hidden files and um I would expect I would I I I assume that a lot of mistakes happen because people will just zip the directory that contains the the project and zip the whole thing. Okay.

[02:35:38]Using something like this. So zip the contents of repres directory pkj and pick up everything that's in there. Very bad idea. That's how you end up with the whole git repository inside the zip file because it will catch hidden files. For Mac users, there's an equivalent of this is to use the finder to zip because the finder the file uh file manager on Mac OS there's an option in the menus to zip a directory. it does more or less this and it catches everything. Okay, so don't do that. Instead, I highly recommend to explicitly mention each and every file that you want to include in uh your uh upload in your in your archive.

[02:36:29]If there's one directory that contains great many files, well, just make sure that there's only the files that you want to include in that directory and nothing else.

[02:36:42]And please script this so that you don't have to redo it each and every time by hand.

[02:36:49]Okay, there are various ways to do it. I there's L3 build nowadays. Uh you can you can use plain old make files, various ways. Just write the command somewhere but don't redo it by hand each and every time. That's where mistakes occur.

[02:37:07]Okay, that's another important aspect is that latte has absolutely no concept of namespace.

[02:37:23]So in a whole distribution like tech live you ideally you cannot have two well no you cannot have two runtime files that share the same name otherwise there will be name clashes. Okay. So uh you should at all cost having um runtimes files that have uh the same name as another one. So the very simple solution you just include the name of the package in the runtime files. Um eventually well just put it in all your files of course except read me. Okay don't don't name the readme file after the name of your package. Okay. Um whenever sensible just put the name of your package everywhere. This is a safe solution.

[02:38:11]Here are a few examples. Well, obviously if we assume that your package is named I'm done, five two minutes.

[02:38:24]>> I I I brought my phone to have a timer, but I forgot to start it. So, [laughter] okay. So, um this one is pretty obviously correct. The second one, say you have a Python script that is used by your package. Well, just put the name of of the package in the Python script.

[02:38:44]Okay. Documentation. Obviously, if each and every package sends a document called doc.pdf or the documentation, you can imagine it'll be difficult to reach your do your your own documentation. Um, this is another one which would not be considered as a runtime file. They could be okay, but it's always safe to put the name of your package inside in the file.

[02:39:07]Okay. Uh, portability. Um well to be portable you should follow uh universal standards obligatory XKCD [laughter] if you haven't seen that one before okay so uh what we mean by standards is basically standards is basically unique standards okay uh so what we ask is that you write the at least the readme file uh in English um as a franophhone I don't really like to say that this is standard but okay. Uh please use u UTF8 uh character encoding.

[02:39:50]Uh so um just line terminators. This is another pretty common uh mistake that we see if you're on Windows. Basically the line terminators can be proper to your operating system. So just use the standard for everybody else. And please don't rely on capitalization of the names to distinguish your package in any way, shape or form. Okay? You cannot expect that on other and another operating system the lowerase uppercase letters will make any difference especially not on Windows.

[02:40:26]Okay. So I have to go a little bit faster. Uh human processing help us help you by conforming to our preferred package structure. This is really coming from from us. What will ease our processing of your uh material? So, we ask for a very flat structure. It's not the time to show us your mastery of the tech directory structure. Okay? Maybe you've read it all. You know how it works. It's not a it's of no interest really for us. Just put it in a flat structure as much as possible. It will be simpler for us. We just ask that there is a top level directory with the package ID as the at the at the very top of your uh zip file.

[02:41:11]Um don't include a TDS.zip file in your upload unless requested by a sitan or a techlab administrator. Okay. So saying someone told me that I should include it is not enough. if that someone is not either Carl or a sitan administrator.

[02:41:34]Okay. So, a short proof very very quickly because I know I'm out of time.

[02:41:38]Uh um I'll host uh te tug in 2027.

[02:41:45][laughter] Uh so this is an example for a package done in literate programming. You see very very flat structure very simple and if it's not done in literate programming so the the style file or the package file will already be extracted you can do it this way okay very simple and if well it's a proof okay so you can see that this satisfies all all the requirements that I mentioned okay package updates if you need to do a package update just go on the page of your package in on on on sitan press the upload button and you will be presented a form where you just have to fill in the new information. So instead of re restarting from scratch each and every time it's much simpler to do it this way and you'll avoid a lot of mistakes.

[02:42:42]Okay, this is uh a place where I just like to take a a minute. Uh so other considerations your automation is our manual work.

[02:42:54]Okay. And I see uh I I mention that especially for very very frequent uploads that are done automatically like well it's a GitHub action. Each and every time that I make it a that I make a commit it it could generate an upload to Satan. It's probably very very simple for you. It makes for a lot of work for us. Okay. So, please take that into consideration. Master the tool, the tools you use. We tend to see that uh lately script kitties that are using the shiny new tools. And when we ask or for something or say, well, this is not correct. The answer can be I don't know how to do it. I don't I'm I'm using GitHub actions and I don't know how to customize it. Well, don't use the tool if you don't know how to customize it.

[02:43:46]Okay.

[02:43:48]Um, your users do not all live on your bleeding edge. Here I'm talking about very very frequent updates. Okay, I'm the type of person who updates tech tech live once per year when they release a new version. So, do you really need three updates per week? I don't think it's that much useful in the tech live world. Okay. So please consider maybe just package a few uh bugs before making an update unless of course if it's very critical. I mean we don't mean we we we don't mind these type of things. And uh when you present a new upload if you have suggestions for the topics that are attached to a package they're always welcome.

[02:44:39]Uh I'll skip AI considerations. That's it's a whole other topic and I'm out of time. Yeah, just one more thing.

[02:44:48]>> Quickly, please.

[02:44:50]>> Yeah, we always reply. Okay, so if you send something an upload, we always reply. So if you don't get a reply for some reason, reach out to us. It's not normal. It's probably that there was a problem somewhere. For example, we're trying to reach you. It doesn't work because you haven't provided the correct email address. Okay. So, we always reply and if in doubt, you can always write to us. Okay. If you have any question, we're there to answer your questions.

[02:45:21]Sorry about the the extra time that I do have someone following you. Sorry about >> uh yeah, I I have a short question from very short.

[02:45:38]>> Yeah. No, we're good to have a question.

[02:45:40]Okay, one question only one question he'll be available to chat.

[02:45:45]>> Yeah.

[02:45:47]Go ahead.

[02:45:48]>> Just a question. Not no answer.

[02:45:50][laughter] >> Okay. The question is uh the user asks whether >> the user asks uh if they have a low and uh if they have a package with a low-level library uh bundle it or have a two separate packages for example if they have tixy uh package and low-level uh PGF library with it.

[02:46:20]It depends. [laughter] >> And the answer is it depends.

[02:46:25]>> Yeah.

[02:46:26]>> Okay.

[02:46:26]>> In that particular case, it would probably be better in one package.

[02:46:30]>> Okay. One package.

[02:46:32]>> So one with maybe just say it's very short remark and I really want to tell it.

[02:46:39]It's very short. I just want to say that 10 years ago at Baja I presented uh a talk where I compared my experience uh submitting packages to C10 cran and CPAN. I was particular and uh my uh conclusion was that the best the most helpful the most warm the most human maintainers were at C10 and now it's 10 years later it didn't change I have experience with us and just wanted to say that we are all so grateful to you and I am especially grateful when you find my temporary files which I usually pollute my uploads and thank you and sorry for being sometimes difficult.

[02:47:27][applause] >> Okay.

[02:47:35]>> I just want to say now >> Yeah. Yeah. And there's probably a moral that uh if you're a seat if you want to host Satan and you're the one and only person supporting it [clears throat] for those of you who've been here a while.

[02:47:58]Okay, enough with that. Uh so are we are we s Okay, so uh we're ready to go on to the next talk which uh certainly has um the most interesting title of the whole conference.

[02:48:14]So you'll have to wait a moment. I'm not going to give the remainder of the title just to keep the mystery going. So the uh this is a remote presentation palanony duza and his talk is entitled a family of strippers.

[02:48:38]Thanks everybody for being here. Um, I'd like to talk today about a program that that's been quite a while in the making.

[02:48:48]Um, I first met this problem at the mid80s when I tried to write unsuccessfully something to separate mathematics from text and uh basically to extract text from a latte file.

[02:49:11]and um and figure out for the first time how hard it would be.

[02:49:17]Uh soon after that u program appeared called DTAC based on wax and um and for quite a while it was an extremely useful program but um latte evolved rather rapidly um from a simple dollar sign and and to separate mathematics from text. We evolved into a number of environments and uh and and the problem became more difficult more difficult and uh and today uh detect um is does not is not able to act on on the simplest files that you that you can feed it. Um I try then to sort of use the same approach which was to list the uh same approach used by detect which was to list all the environments and how to treat each one of them. You know, if you want to extract the mathematics inside it, if you want to extract the text inside the mathematics that is inside an environment and um and and deal with each one of um the cases um and um probably using some more modern tools for doing the parsing. Um there's a lot of modern tools today's to do parsing of uh programs stuff which is way better than what we had back then which was lax and and the the tools that accompanied and uh but even then you know we don't have a full tag parser independently of tac itself and um and This talk is about you know the uh the challenges of using a pseudo parser. You know something that parses part of um of the text of a a latte file and using it in order to extract mathematics and extract text so that it can be analyzed separately um as one may need it. Um the uses for it are obvious you know you we can my first use was um spelling grammar tracking you know and uh but today machine translation of technical books and and full text search you know training of large language models uh accessibility uh it comes um as one of the requirements that that one really need to know the boundary of uh mathematics in in text within a within a a PDF file within a a source file for that for that matter.

[02:52:32]Latte is a programming language not a markup language. So if you look from that u from that view u any tool that reads a tech file without running tech is is basically writing its own interpreter.

[02:52:50]Um, and that interpreter is bound to be incomplete unless you put a lot of time into it. A lot of time into it.

[02:53:00]Um, what makes it so hard? Um it's you know the language is um I mean first of all we have some um a problem which a language which is too incomplete m has contains a tour incomplete microsystem and um userdefined commands everywhere these days. You know, the files are extremely more complex today than they used to be. And um optional arguments, stereo variants, you know, cat codes, meet document, verbating environments, um mass notation versus math pros, and on top of it all, we have the multiple language per document problem. um in science for example, it's extremely common not to um not just in citations during the text but uh on um bibliographies that extend over several different languages including right to left scripts, CJK uh language that have liatures which are specific to the language and um and even fonts that have um discretionary liatures which are specific certificate to the particular font. So that complexifies the problem um even further. So my idea was to um I I I I told you that there are some very developed u u tools for doing parsing these days. Um and uh but I shy away from dealing with them and uh and try to use ready projects which have been tested and established in in terms of parsing tech and I choose the four that I consider to be the top uh ones which is Latte ML uh Latte Tom Tech Soup and Pandock and then use them as a module inside um uh a program almost in a way that you could replace one with the other using them just to parse and to provide the tree with a separation between text and mathematics and uh and you know also to experiment to see which ones would provide the best results.

[02:55:31]uh latte ML was uh the first one. It's it's an ambitious approach. We reimplement ST expansion engine in Pearl and it produces XML, MathML, HTML, handles, you know, almost everything that uh you would have on a normally um on the normal um file on a standard file a latte. It deals with a large percentage of the files which are contained in the archive for example and it's also an active development supported by NIST and AMS and a few others um for text extraction is um is very accurate for well behaved documents um and uh and allows the mathematics to become MathML. Uh the limits are that is slow. It has a very complex it's not part of tech live. Um it fails on on certain macros and u and it has no multilingual text separation. uh on text separation, you do want uh each language to be separated in each in in an independent file and if possible for those files to be tagged as BCP 47 so that you can easily feed into um uh other tools like language tool or spellers for the particular languages and proceed with uh with analysis. Um none of that is possible with latte ML.

[02:57:01]Um, latextom is a has a parser which is directed to certain specific um and limited um uh commands. Um so it if you want to for example obtain metadata out of a file u you know the author you know the title the pageionation you know the address of the authors and so forth. It's it's a it's an exceedingly uh good program and uh but in order to parse a general latte file all the way down to the end and and do extraction at every change in between mathematics and text it's it's it lacks um and is an incomplete parser uh by design. It's not it's not uh built to parse the whole document. is built to parse only um certain specific parts of the document that you may need ahead of time. Um it has f is fragile and complex macros absolutely no language awareness and is minimally maintained these days.

[02:58:11]Um uh so it you you even uh end up with um commands which are unrecognized inside you know uh text uh nodes which is undesirable.

[02:58:28]Then the next one uh that I tackled was tech soup. Um it's one of the first times that Python met Latte uh very successfully and um this is built on sort of uh the same ideas of beautiful soup and I guess even the same framework and um and it parses tech into a navigable tree which is really really nice and allows you to sort of pick up each one of the elements and analyze it for mathematics and for text and separate them. Handles balanced braces, optional arguments um and is good for scraping content and this and that's mostly used for scraping contents. The limits of techup are that there's no micro expansion.

[02:59:20]uh breaks on very simple stuff like verbating and listings, absolutely no language separation and um fails on very simple unbalanced braces and uh and most of the use of it is is as I said for um for text extraction for scraping content out of um latte files. Uh and then the next one that I approached was Pandock.

[02:59:51]And Pandock has a different um approach in um it is a most of you know already what Pandock is, but Pandock is a is a is a Swiss knife tool that is allows you to change uh formats among a humongous amount of formats and uh it reads the latte tree and and you're able to even extract plain tech out of it right away. Um, but most beautifully you can insert some markers on it with a pre filter and then uh establish what you want it to do with certain environments that it doesn't know about and then uh and later with a with another uh with a processor itself.

[03:00:43]um extract what you want um out in text and what you want out in um in mathematics and uh and deal with them appropriately.

[03:00:57]Um this if you compare them you you can get an idea right away. Um, Latte ML is the second most capable of one and um, Pandock is um is is um, very uh, very uh, able to deal with it because it doesn't require any um, none of those require any uh, tech run. uh they they run by themselves within their own environment and uh and Pandock is extremely fast and is able to do the separation uh uh very well. None of them uh handles multilingual liatures for example. And um and the way that we chose to build this is almost in a way that almost in a way that you can um uh use this as a plug-in to the main program. So you can basically choose to use Latte ML, Latte Tom, Tech Soup or Pendoc and and and run to get your extraction. Um so um how does this work? It works very much in the same way that the old DTAC work. Um some rejects, some heristics um and uh marking up inserting markups in uh in in the text when when you want things to be um uh uh be sorted out in a different way. and then picking them up, picking them up later. Um, listing the environments that you need specific treatment. Some environments are scraped altogether.

[03:02:48]Some environments are searched for text within math mode and uh some environments are you know discarded environment and use the full content of the environment uh in a way that you can um update this list as you go. So that if you write your own environments for example, you can add this to a list that will be searched and uh and that will be uh used for the extraction. Um the the four the you know the pro stripper uh the first one is um literally you know the core idea is to use rejax you know parse it off and uh and try to um tell them which environments you want and which environments you don't. Um the Python strip is a step forward from that but um you know it it almost um works exactly in the same way. Um a dictionary must be maintained with the list of um the environments you know and how you want it you know it to be dealt with. Um but it contains no font literature awareness. Um Matthew Handley is very crude. Um uh polylossia uh presents its own problems and uh and and it it is still a very hard problem to solve though much better than anything that you can get out of DTAC.

[03:04:26]Um the fundamental limit of this whole thing is that um they parse all four of them parse the source of a latte document. But what we want is the text of a compiled document and these are not the same thing. Um, latte is a programming language and you can literally uh create a a file that uh at every turn every turn of a processing of it by latte it will produce a different output. It will change itself in order to produce a different output.

[03:05:06]Um, so custom macros, input chains, got code changes, fing code liatures, language switches from packages, all of this makes sense after tech has run but not in the source that that you may have at your hands.

[03:05:24]So with a pendoc stripper you know we we we try to start with um with the latte go through a pref filter that do the micro expansion you know and marks the languages before you do any further processing and then uh was a simplified tag you know we we we would take the markers and analyze them and separate them and uh and deal with them uh in um in other separation of languages etc. This the only one that does separation of languages in a sort of a a nicer way um among the four in the set.

[03:06:11]Um the prefilter you know of pandock scans the preamble for a new command and does the macro expansion. Um I think it takes only uses only four levels. Yes, only four levels. Um it detects language declarations and use BCP 47 tag map in order to uh mark them up and um and uh injects certain tokens at language switches so that it can be later dealt with uh when you get a sort of a JSON file or you get a tree that you can traverse and separate them and uh and strips the noise out like you things like label, to-dos, and include graphics and so forth. And then um and sends it out to be to be processed. And um the we have you know um a math filter that also allows you to to deal with the mathematics or to just put a um a marker that that area contains some mathematics in your your new file and also to deal with the issues of text within the mathematics which also is extracted so that you can deal with it.

[03:07:35]Um the um the good of it is that um you know citation handling is um is performed at a at a at a very high level. you know, it's able to understand uh um certain languages of certain um of citations which are uh prepared appropriately and uh and then you can also you can send them separate those later for uh for text extraction.

[03:08:11]Um if knowing these limitations and knowing that we wouldn't be able to go much further with a lot of this um uh using those uh four uh pre-processors.

[03:08:29]Um I tried another idea which was uh to create a stripper that would be parsed by um by Latte itself by tech itself um basically to be uh processed by Lu uh and uh and allow Lua latte to to do with the text separation and the tagging and the and the liature handling and so forth And um I kind of stole the idea from a package called um spellings um which is um it's in the repository and it's it's it's part of C10 uh not very usable these days because of lack of maintenance but that it does use this idea in order to do spelling of um of a a source tech file.

[03:09:28]>> [snorts] >> And um so um the the idea was it it looked good in the beginning you know uh it it would have zero overhead if work if it could be built. Um custom micro expansion wouldn't be partial like the other cases. It would be full. Uh language separation will be exact and um font liature could be dealt with and uh footnotes could be dealt with. Uh uh noisy command stripping you know and uh site command handling speed would be zero overhead since you one would probably running low attack already and um and the only problem is is that you need processing. Um so um I settled on using a pontto stripper when you know no full build environment were available. quick construction without compiling and uh and um and standard tech and common packages were used and uh and then using LUT attack stripper which was a little bit more evolved using uh when I need exact language attributions when I need uh custom font with liatures um and I had a file which already compiled under the royal attack um tech obviously you know provides a whole environment embeds lu 5.3 interpreter inside and uh and he text build a node list for each paragraph um the tree that that is um that describes the text and the mathematics is readily available in there so one can easily traverse it and um and uh end up obtaining the information about text and mathematics So we'll start it out with a tech source, go through a lu attack macro expansion. Um get a node list um and uh and then with a Lua walker we go into this node list to obtain uh the particular extraction and then do the output on a per language field and in text and in JSON.

[03:11:51]Um it has a barbell track a polyosia track. So it has no information about BCP 47 uh languages inside. uh it uses the information uh which is inserted there by Babel and by Poly Glossia and uh which makes it you know one of the first fully polylossia ready uh fully um BCP 47 ready uh programs um in our multil- language environment lots of problems in dealing with it you know u there are lu uh lu has two renderers you um one is the default render in Lua, the other one is half bars. They deal differently with um with liatures and um so we have to um expect both and then um and uh and then uh proceed uh to analyze how to do uh liature handling for each one of those. Um and then you know we We had to deal with a few other things like language of boundary space and footnote extraction you know and uh mathematics is is once it's recognized it's marked up in the text as mathematics or display mathematics and uh and um biblotch integration is is is fully ready you know you you don't need to do anything besides marking up your your uh your text for for um for languages and uh and you're ready to do this. You you just add use packages stripper in a file run blue tech and then you get you know from my book.te you can get uh my book en- us annotated.ext my book extract JSON my book fr.ex my book ja.x text for Japanese and so forth and have every single piece of text separated by language on the other side with a parsing which is completely done by by by tech.

[03:14:07]Um the two groups side by side uh the stripper the luite tech stripper is a is a clear winner you know um by dealing with uh custom macros math handling you know uh multil- language out multilingual output babel support polyos support BCP 47 native and uh dealing with liatures and footnotes and uh bibl citations with zero overhead had is a clear winner. You can get access to it by um going to my uh my GitHub page uh pa lun p a u l o n e y and uh downloading all four um sa handler strippers plus um the stripper package um under lua tank and um and I hope you enjoy try um try it out and uh send us some lists of um problems or wishes that you may have. Um thanks very much for listening.

[03:15:38]Yes, I do have the time.

[03:15:40]>> Oh, okay.

[03:15:44]>> I just want to say I'm sorry for the malfunctioning of of the slide uh of the slides during during the recording. You can get you know the full set of slides at the at the website for the conference.

[03:15:58]>> Hi Paulo, no planning here. Thanks for your presentation. Um I'm always happy to see new text extraction features because we are fighting a lot in archive because we get a lot of we have to text extract and do text overlap and this kind of stuff. Have you compared to PDF minus 6 which is one of the standard extraction algorithms. This is the one question and the other is how does it deal with these nasty little PDFs that contain like spirals and whatever other things. Um, so that you can can always work on Latte only. Does your code do anything on PDFs by itself or only start from Latte?

[03:16:39][snorts] >> Well, well, we may not have time for all the questions in there. There's like five or six questions there.

[03:16:47]Yes, we did compare that with PDFs and and uh uh the coverage on the four strippers, you know, the uh the beautiful soup um you know uh tech soup and then and latex ML and so forth. The coverage does vary and uh but it's is superior to do anything on the PDF. It's superior and uh but but nothing compared to the to to the stripper written for lu attack luat because that that is 100% accurate you know it's um is basically you say I want math I want text and you get math you get text and if you have your bibliography in like you know 20 languages in your bibliography you get all of that separated in 20 different files marked up by BCP 47 and everything is ready Um I I will try to address some of your other questions. What was the other question now for me? So we do normally text extraction from PDF even if we start from a later document that is not optimal because well we know there are a lot of problems there. Um how good does it compare to PDF extraction from PDF when you run it uh over the latest source? um considering that we it might have a lot of complicated macros and what we we get we get files that have every single word replaced with a macro and so um I I'm not sure how good it deals with that.

[03:18:21]>> Yeah. Well, it there is there's a divider here. you know there's there are force strippers which are which are written for pursue the parsers and and and yes indeed you know complicated macros um botch things and uh so um latte mail is is changing right now and uh so we'll change accordingly and and and hopefully you know um will still cover most of the files but yes if you have complicated macros and you if you have complicated stuff it will budget those um the lu attack stripper is is is 100% sure you know there's absolutely you can do spyros you can you can make your your text to snakes on tixie and uh and it will it will take those and serve them back to you as it is >> great thanks >> thank you >> hi Hi, M Krueger here. I'm wondering you I mentioned that you have this handing for liatures with the lure renderer and the half bus renderer and it seems like that is a bit unnecessary if you just hook in a bit earlier and just extract the text before the font shaping is happening. Did you consider doing that?

[03:19:48]Well, I have to take whatever Lu Tech can offer me and I and I have to tell you that uh you know I'm not uh you know I'm a mathematician and and not uh uh uh a very well-versed tech programmer and so I may be missing something that would be easier to deal with um as you parse your you know the um the word differentiable over you know over Lu Lu you get, you know, depending on the font that you have, you may get a liature.

[03:20:25]Depending on the font that you have, you may not get a liature.

[03:20:29]And so recovery from liatures is a big part of of the low tech stripper right now. But uh I would be I would be ready to listen to other ideas to deal with um that don't have to read directly from from the latex source. I mean reading from the latex source is sort of a a nogo because because of the fact that latte is is not a markup language. Latte is a programming language. You know, we're interested in in the text that shows up in the PDF. [clears throat] You know, I I can write a latte file that uh requires an infinite amount of processing in order to get the results that I need, the results that I expect.

[03:21:22]And uh so analyzing you know uh after the parsing by by lu attack is is was the objective of the Louis stripper but uh I'll write you and uh and I want to listen a little bit more careful to what your ideas are for um not having to deal with liatures because that that will simplify the parser tremendously.

[03:21:49]>> Thank you.

[03:21:51]>> Okay. Perhaps we could thank the speaker again.

[03:21:55][applause] Very nice.

[03:22:00]>> Thank you very much.

[03:22:11]>> Hi.

[03:22:14]>> Okay. Can you help me get correctly combodulated >> like this?

[03:22:21]>> Yeah.

[03:22:22]>> Just a minute. Oh, yeah.

[03:22:24]>> Very well.

[03:22:27]>> And then Oh, huzzah.

[03:22:31]Are you okay?

[03:22:31]>> And then just wait. We'll get to >> No problem.

[03:22:35]Um, while this transition uh happens, I simply have a a tongue-in-cheek question for the conference organizers. If scheduling a talk called a family of strippers right before the Catholic theologians was a form of hazing.

[03:22:51][laughter] >> Coincidence.

[03:22:54]>> Yeah. All right. Understood. Very well.

[03:23:00]>> So, do you want to >> I I'll introduce myself so when you're when you're ready for me.

[03:23:06]>> Introduce yourself by all means.

[03:23:07]>> Oh, I think that'll just distract. Is this okay for folks in the back?

[03:23:10]>> I think the mic will help.

[03:23:12]>> You think so?

[03:23:14]Okay, take a chance. Thank you.

[03:23:16]>> Is this okay? All right. Very well.

[03:23:22]>> All right. Good afternoon everyone. Uh this is my first time at any tech users group conference. Thus far it has been a lovely experience and everyone has been deeply welcoming. I applaud in a special way Max Chernoff as the local organizer.

[03:23:40]This venue has been wonderful. Being right along the riverfront has afforded me the opportunity to engage in long outdoor exercise in the fresh air that feels much better than the air back in Houston first thing in the mornings. And Carl Barry was most communicative and helpful in uh making sure that I was able to make it here to speak.

[03:24:04]Uh my kind of recurring internal theme through this conference has been rethinking and reconsideration.

[03:24:12]Yesterday afternoon my dear deskmate for the duration of this conference, Professor John Leinhard of MIT made me think seriously about long form assignment submission and styling requirements for my undergraduate and graduate students. Yesterday morning, Pavit Aurora inspired me to reconsider aspects of Zettoen and my academic notes to manuscript writing and research workflow. And of course, Friday midday, Timothy Ericson and Aruat, I'm sure, inspired basically all of us who work in education in some capacity to reflect deeply on what accessibility in the materials that we present really is. and I give to them the highest of praise and encourage anybody who might be tuning in online to revisit that talk from Friday.

[03:25:09]There have of course been other talks during which I have scribbled ideas about Latte to revisit or reconsider in the near future uh in my digital notepad. In a way, my own talk is about revisiting or reconsideration.

[03:25:25]I work in a general academic area as I'll discuss in a moment in which the vast majority of my colleagues are simply unexposed to law tech. We do not work in a law tech realm. We work in a word processor realm. I've prepared this talk accordingly with two audiences in mind. The first is everyone here at the worldwide tech users group 2026 conference in Calgary. maestros and maestros of tech and lawte. The second is colleagues who are uh might be positioned to tune into this online, maybe at a later date, check in on YouTube or some such thing, but are totally unfamiliar with Latte. So, with that setup, I hope that will explain for all of you why many of my slides seem to be communicating at two levels. That's because I am indeed intentionally communicating at two levels.

[03:26:24]Is it arrow down?

[03:26:25]>> Yeah.

[03:26:25]>> Thank you.

[03:26:28]>> All right. I serve as lecturer in religious studies at the University of Houston, course developer and instructor in the PhD program, theologia, doctor, doctor of theology at St. Leo University and instructor of Catholic theology in the core curriculum at the University of St. Thomas at Texas. Broadly I study petristics that is the writings of the early church fathers and the theology of the Latin middle ages Greek Bzansia. My research topics include Greek Petristic Christologology understanding of who Jesus is basically and cosmology understanding of the structure of all that exists things visible and things invisible plus their reception among Latin scholastics in the Middle Ages.

[03:27:11]So, I'm the author of a brand new historical monograph published by the Pontipical Institute of Medieval Studies here in Canada, but way over in Ontario, called Maximus, the Confessor's to Mystic Legacy. So, what on earth am I doing at TUG 2026?

[03:27:28]I have a kind of parallel set of interests that are somewhat hobby and somewhat professional. A kind of perfect blend of the two in which I advocate software freedom in academia throughout academia, teaching, researching and so on by delivering talks like this one. So that's this fits within that kind of second portfolio as it were. For more about anything about me, go to coreystephin.com, which also links to my gopher hole. That's for the latte tier of my talk. Um, that mirrors most of the material there in.

[03:28:07]To start thinking about LTE for humanities or divinities in my case, disciplines uh in general, we have to think about LTE beyond its normal bounds. The world's best established software system for type setting and document formatting. Watt has been popular throughout mathematical and physical sciences for decades. As plainly evidenced by the fact that many of you folks have decades on me in terms of years on this earth and have known each other and worked with each other for longer than I have been alive.

[03:28:39]The utility of lawte for humanities disciplines however tends to be overlooked as evidenced by the fact that I doubt that any but one colleague that I know most broadly understood actually uses latte in any professional capacity.

[03:28:57]At the same time, I contend that Watt can be used for type setting in most instances in which a scholar of humanities needs to present her materials.

[03:29:07]And this is for that online tier, right?

[03:29:10]I prepared these slides in LTE in fact with less less frustration than I would have had with a graphical presentation program as one file. And uh there on my GitHub repository uh historical theology latte talks a person may view the source for these slides and also a preprint I believe it's called for the subsequent tugboat entry.

[03:29:35]In today's tutorial, my aim is to introduce attendees at both tiers to the use of law tech for a common task in humanities scholarship, namely type setting texts with unusual line length and other organizational needs such as primary texts with accompanying translations.

[03:29:57]The thought experiment for the day in which I ask all of you to join me will be that of determining how to prepare a handout for a historical conference or class presentation with perfectly organized excerpts of a Byzantine Greek text and an original English translation placed in parallel.

[03:30:20]Of course, before we get into that, we need to think about why somebody in my areas of work, as most broadly understood, might choose Lattk at all.

[03:30:29]In humanities, or in my case, divinities disciplines such as my own area of historical theology, editors expect us to prepare our materials for publications with bug forbug, and I chose that very intentionally, compatibility with Microsoft Word for back and forth markup. In practice, this expectation requires the use of word processors. Don't rely on man pandock and magic sauce when somebody's expecting bug forbug compatibility. So I use Libre Office Writer for example prepare according to an editor's and presses specifications send to the editor. Editor writes comments and marks sends it back to me. I reply to those and insert in lines back and forth back and forth and the same docx file grows into this vast cobweb that eventually becomes god willing a publication.

[03:31:23]A word processor like lib office writer presents material in a what you see is what you get style or wizzywig as we say. While one writes why one is simultaneously thinking about and working on formatting and so it is for that second audience the audience online. I say, "How many times have we gone, "Oh, fooy, this footnote got messed up in its alignment. Now I need to spend the next however ever many however many minutes not thinking about my work itself, but rather fussing about the editor's exact specifications for a format." And then poof, the train of thought is lost. The break between classes comes to an end, and one is just frustrated.

[03:32:08]Enter the type setting language. In a type setting language such as such as latte or cont or typed, either of which a humanity scholar might also choose for various reasons, though I'll talk a little bit more about that later. The stuff we're talking about today is a really latte exclusive. One is only concerned about the structure of what one is writing. One declares that specific characters or items will be in certain categories of writing things.

[03:32:37]Boy, explaining this in front of Mittbach and Fischer is making me a bit self-conscious. One declares that specific this is how Watt works. One declares that specific characters will be in certain categories of writing things heading, paragraph, footnote, accent type, type face, etc. and one allows the program to compile one's writing that is really a kind of code into a tidy formal document nearly every time preparing a curriculum vite which is actually what prompted me to learn law tech I grew tired of wrangling a curriculum vite in a word processor and look for something better or a conference handout like I'm discussing today or another document with unusual formatting requirements becomes about the writing itself rather about fussing about all the little odds and ends of the formatting.

[03:33:30]Of course, there's a real investment of time in the front end, but then you just get to sit down and write to learn law tech as a scholar in the humanities. I actually think that it's worthwhile to make a couple of specific recommendations.

[03:33:45]Here are the two that I recommend. The first is what I actually used to learn tech, the notso short introduction to law tech. This is available in this GitHub repository.

[03:33:57]The second is what my one colleague I know who also uses latte learn to learn latte used to learn latte modern law tech and that's available available in this GitHub repository.

[03:34:10]Either guidebook may be used to begin to learn how to type set pros, letters, ancient languages, and other materials that are relevant across the humanities without the strong frontloading of mathematics and physical sciences that one finds in other introduction introductions to Latte. Either option, in other words, provides an efficient introduction to Latte for a scholar in humanities discipline. One of the beauties of Watt is that it's so vast with so many different pathways down which one can explore.

[03:34:41]But one does not actually have to learn all those pathways. One can just learn a few specific packages or tools that one needs within the broader ecosystem and make use of those.

[03:34:56]All right. Now let's pivot to the whole handout thing.

[03:35:00]There are many ways to set parallel text in Latte. The simplest or at least the simplest of which I am aware is to use the community package called parallel which is included in techlide mtech and other latte distributions. For full documentation, visit ctan.org/pkg/parallel.

[03:35:20]To start using parallel in any latte document, include the following in the preamble. Use package parallel. Now, why am I saying this at the tug? Well, for a newcomer to law tech, it is extremely easy to prepare an entire document only to have the whole thing fail and really struggle to figure out that the problem was just forgetting that pesky line in the preamble. So, I'm going to reiterate that a couple of times for the benefit of that second audience, the online one.

[03:35:52]For example, newcomers, here is an abbreviation of the preamble to this beamer that is latte presentation file with the most important entries including well parallel because I'm using parallel in this talk. The document class is beamer that is a presentation with that very important aspect ratio 16 to9 which was required by the tug folks use theme goods and uh that's basically what this theme is a built-in beamer theme. I use a custom type face because I'm showing uh polyonic Greek later on. But uh use theme gutsing will get you 90% of the way there.

[03:36:30]Ellipsies somewhere in the middle of all that I've included use package parallel.

[03:36:35]And then finally a nifty title with a couple of line breaks and a hyperlink so that on every slide everything just automatically shows up in what I consider to be a neat setup.

[03:36:51]To begin using the parallel package after declaring it in the preamble, the basic format is well quite basic. Begin parallel text left space adjustment if needed. Right space adjustment if needed. Then on a line a new line parallel L text left text row one. New line parallel R text right text row one.

[03:37:12]Parallel par triggers a line break. And then parallel L text left text row two.

[03:37:17]and so on and so on all the way down until the end of the parallel text.

[03:37:24]An extremely uh basic example then of something that I might set in parallel.

[03:37:30]Uh the start of the traditional Latin prayer, the A Maria begin parallel with no uh uh spacing changes for the columns. Parallel L text a Maria parallel right text Hail Mary. Parallel PAR to trigger a line break. parallel L text Gratzia planer parallel R text full of grace and then I've just inserted some ellipses here and parallel and that's how all of that looks at the bottom first two lines perfectly set left to right with equal spacing um and all of that no fussing with word processor tables here folks for advanced setting of parallel text in Lothk especially especially long- form setting like a critical edition of an ancient text with accompanying translation, one really ought to learn the two interconnected and frequently updated community packages. ReLEDP par for parallel type setting specifically and reled Mac for type setting academic additions in general. It includes all sorts of features like automatically uh configuring a critical apparatus which is extremely an extremely powerful thing to do and basically impossible in say a word processor for example. Both of these relay par and relay mac packages are bundled in techlive and other latte distributions.

[03:39:00]Rel Mac are purpose-built for long for academic parallel texts.

[03:39:07]Their functionality actually at this moment in time cannot be duplicated in any software of which I am aware. So I specifically looked into can you do anything like this say for example in types and the answer is a categoric no.

[03:39:22]In fact types basically can't do parallel text period. I looked uh you know can you do this again I mentioned the word processor the answer is no and so on and so on to achieve this uh setting a critical addition for oneself say to make a handout which would be the context for doing that sort of thing the example to have it turn out cleanly is basically to learn learn law tech and then learn reidpar and reid back these two packages it's important to note are meant to replace older methods of academic parallel text preparation ation in LTE than one might find in web searches. The most obvious example of this is if you cut out the re from both of them, the older leadpar and lead mac.

[03:40:05]No more use reled par and reled mac which are total rewrites with vast expansions of functionality including full unicode support working with lua law and so on and so on.

[03:40:19]For full documentation go to ctan.org or the package page for ReLED par and the package page for reel reled Mac. And I'm going to hit this in again for newcomers that second tier viewing online.

[03:40:30]Remember in the preamble, lest everything should fall apart.

[03:40:39]The most important packages for handling text outside of Latin alphabets are the community packages Babel and Polylosia which are included in Techlive and other Latte distributions.

[03:40:52]Although it is possible to convert a project from one of these let's say meta packages to uh to the other. Oh, that typo will be fixed in the [laughter] >> in the Tugboat article. one ought to invest the time to choose carefully between either of these two at the start of a long form project. They have similar um uh what's the term I'm looking for styling or or uh uh code code uh words, let's say. Uh but they're not identical.

[03:41:23]So, it's not trivial, not terribly burdensome somewhere in the middle. But if you're starting a long form academic project that will take years, just take the time to figure out which one to use at the front end.

[03:41:37]How do we choose between Babel and Polylosia? On the one hand, it might look very easy. uh and in the latte companion third edition published very recently 2023 Frank Mittbach and Urria Fisher both of whom are fellow speakers here at TUD 2026 suggests despite that despite the fact that Babel is older than polylossia Babel is generally preferable quote unless you are in a special situation where a language or script is notably better supported by polylossia the recommendation is to use Babel with all engines of course in my areas of That's a big unless since now Babel and Polyia technically support the exact same set of languages.

[03:42:23]But a situation in which uh say an ancient language is better supported than polylossia in polylossia than in babel is much more common than in a modern language. That's where it becomes very important for one of my colleagues to go read the documentation for both and figure out okay what one which one do I need right now this situation no doubt will change over time but that is the uh uh state of affairs in 2026 now in old web forum discussions one might read that babel will work better with lua latte whereas Paul Gissio will work better with zilk this is why we don't go read old web forum posts in 2026. That is wildly incorrect. One could choose either Babel or Polylosia for either of the two mainline engines that are built with full Unicode support, Zlatt or Lua Latte. Of course, results might vary in compilation on either of these engines depending on the specifics of a given project to bake it for choice.

[03:43:32]Where we need to go right now in the kinds of work that I'm in is just straight to the official documentation, read it, and figure out which of these is going to work better for my project and possibly even do a trial run with a short amount of text. Frankly, now learning Babel, well, here I'll earn some brownie points. I recommend chapter 13 localizing documents in the latte companion third edition by Melbach and Fischer. This chapter succinctly introduces localization in Latte via Babel in a practical dedactic teaching style. So that's pages 297 to 342 in the second volume of the lot companion third edition. Um, believe it or not, the lot companion is the thickest full set of books that I own, even as a historical scholar with a wide variety of lexica.

[03:44:26]Um, so, uh, take that for what it is.

[03:44:30]Okay. Um, it's even thicker than my petristic Greek lexicon, which is some 1500 pages. So, yes.

[03:44:39]Now to learn polylossia I recommend the packages official documentation and that is on can uh pkg polylossia.

[03:44:50]Okay do as I say not as I do. Here I will model polyia which is just what I happen to be familiar with using for polyonic Greek ancient and Byzantine Greek. I read the entry in uh the Mitovak and Fischer book uh the day before that submitting these slides was due. So I did not have time to learn a whole new language package. But after reading that recommendation, I'm I certainly intend to try Babel for a future project and see how it works.

[03:45:20]To use the package Paul Glossia, remember to include it in the preamble.

[03:45:25]Then after the beginning of the document but before the actual type declare the languages and their variants. For example, here is this.te files Paul Glossia declarations. So the declarations I needed to make this beamer presentation. Begin document set default language to English. Set other language to Greek with the variant ancient. And that allows that variant ancient is very important. It's important to read the documentation to see what variant options there are. For example, a German speaker might need Austrian or Hulk Deutsch or some other option to be selected. In the case of Greek, um none of the uh polyonic marks, the breathing marks or the accent marks would be able to appear without choosing that option and hyphenation would be a disaster. But with this option selected, hyphenation and breathing uh excuse me, accent marks work. uh breathing marks work and hyphenation mostly works.

[03:46:26]Now, when using one of the other declared languages, toggle it when needed. A language may be toggled for a long excerpt like so, begin language, then the languages text, and so on and so on until the end of the language. For example, begin Greek and then here I'm using uh John chapter 1 verse one as an example from the Christian New Testament. And in the tugboat article, I'm also going to use the start of Homer's The Odyssey as an example.

[03:46:54]Um, so you see begin Greek, then a line with a line break, another line, line break, another line, and so on, and Greek. By the way, here and in other code samples, the Greek text is spaced incorrectly due to a limitation of the package LST listing, which I've learned in this conference is basically becoming obsolete. But no, it's not becoming obsolete. Okay, in any event, there's a limitation with the way that it shows uh spacing in non-Latin languages. And so uh that's that's where we get this result. Um if somebody wants to talk to me after the talk about a potential fix for that, I would be all ears.

[03:47:32]In any event, the result of all this is this very clean, lovely uh portrayal of John chapter 1 vers1 and arai in hologos in proton kos in hologos.

[03:47:47]Now a language uh uh may be toggled for a single line like so with polylossia text language text in language for example text Greek Greek text. This might be used to insert a keyword or a short phrase or even an inline quotation for example hologos produced with text greet hologos inside running pros.

[03:48:13]Now to combine parallel in poly glossia for just about the simplest setting of parallel text that I've been able to figure out to achieve in latte begin parallel and then determine if you need uh column width adjustments text language parallel l text ancient text line one parallel R text modern text line one parallel par to trigger a new line text language parallel a text ancient text uh two parallel R text modern text line two and so on and so on all the way until the end of the parallel.

[03:48:52]Of course, this becomes wildly tedious very quickly for anything more than uh certainly more than a few stanzas of poetry or some such thing. I might even say more than a few lines. And this is why I remind everyone of reled PAR and ReLED MAC for anything more complicated uh and or significantly longer in making parallel text displays.

[03:49:19]Of course, here what I'm going to show is the simplest example because the simplest example is probably the best way to get started and also what most people will want in preparing a simple handout. Right? The idea in a handout is show as little as is needed to illustrate a thing. And so here I'm showing similarly as little as is needed to illustrate the thing. Here is a simple example of parallel and polylossia working in combination to display uh how I might show on a handout my own translation from the opening chapter of my new historical monograph maximus the confessor's to myistic legacy. So you see begin parallel uh zero uh spacing left zero spacing right text Greek parallel L text then the first line of Greek parallel R text the first line of English parallel par to trigger a line break text Greek the first line in the Greek parallel R text the excuse me the second line of Greek parallel R text that second line of English and so on and so on down to the end of the parallel inserting this into a conference or classroom handout the result might look like the Next slide.

[03:50:27]So here I'm not going to read the Greek nor the translation to you folks. That's not the point. Rather, I want to point out a few really uh compelling things here. First, I'm using the koka type face which is specifically designed such that Latin and uh Greek look how Latin Latin alphabet and Greek alphabet are supposed to look but with equal spacing.

[03:50:53]and this kind of consistent flow. And so that's again that's Koka.

[03:50:58]And so there's this really neat flow in which the tech the Greek polyonic Greek looks like polyonic Greek. It has a kind of homage to a Byzantine manuscript or some such thing. And the Latin text looks right. It looks like Latin alphabet text how we expect in modern English. And yet the type face is consistent, right? And so that's a uh that's not necessarily special to Law Tech, but it it's compelling that there are options like this built into Tech Live is what I want to say, right? A person doesn't need to go hunt down new things. Just install Techlive and boom, there it is.

[03:51:42]Maybe more seriously thinking about something Latte specific.

[03:51:47]Notice how all of the spacing just works. Earlier in the talk, I said no messing with word processor tables here.

[03:51:57]I'm in a discipline in which that is by far the most common way to set parallel text. In fact, it's what's expected by most presses for submissions of parallel text. And good heavens is that ever a frustrating way to handle parallel text.

[03:52:12]Everything breaks all the time. It's very difficult to keep track even in one's own writing what is supposed to be in parallel and so on. Here every single time even when there are page interruptions and things of the kind everything is just right.

[03:52:31]I close with a couple of notes and I know that everybody wants to eat lunch uh such as the curse of having the pre-launch time talk.

[03:52:42]Scholars in my disciplines getting started with law tech here are some things that you folks watching online might want to consider. First look into bibl and to appease Mr. Carl Barry I'll mention bibtech as well. a comprehensive citation and bibliography management system that is capable of being interlin with a variety of citation managers including like probably a a much higher percentage of persons in my discipline are using rather than lawtech zotterero and also jabref which is extremely common say among legal scholars.

[03:53:18]I also recommend tech studio and other law specific graphical writing environments. Latte of course can be written in any plain text editor. I've seen a few of you folks with your terminals open to Vim or Emacs.

[03:53:30]Debates on those two can continue after this talk.

[03:53:33]>> No, there is no debate.

[03:53:35]>> Okay, [laughter] but I'm going to suggest that you know especially for a newcomer, a designated environment can help improve efficiency and avoid code errors. And I love Tech Studio for about a million options. I've tried a few different uh software solutions. First, Tech Studio is entirely free and open source software, which is great um in so many ways. It's all locally installed, integrates neatly with the most recent Techlive installation. Um it's cute based, the cute uh toolkit for theming. And so in my uh GNU Linux based setup, uh I set cute. Right now, I'm I'm in love with Dracula, and I get that sweet uh vampiric darkness uh whenever I'm writing, right? Uh here, all these brownie points. So, great. Um but, uh it also has some great features like it can predict all of the most common latte commands that I might want to trigger.

[03:54:31]And it a as somebody who's a relative newcomer compared to most the people gathered here, having having an autocorrection for some command that I don't trigger very much is just very helpful. Another reason that I love it that I haven't been able to use yet, but this is more of just an announcement of a feature for everyone because it's quite new to Tech Studio is there's a very easy self-hosted collaborative editing option. So a person doesn't have to rely on a paid proprietary online system but rather can just in install tech studio have one's colleague install tech studio um I think you just need to port forward and to click go and and then it just works and so that's a really compelling thing to get used to using tech studio and then if you want to coll if you're in a discipline in which people actually collaborate in law tech which I am not but uh you can take right you can make use of that. I also recommend colleagues look into designated packages for individual languages. So my one colleague I have who uses Latte is a Syriaist and part of why he uses LTE is there's this super powerful grand package called Sudo Syriak uh formalized Aramaic. So it it allows a whole bunch of different type faces that are specifically written to show Aramaic in ways that people who use Aramaic Syriak expect it to look right.

[03:55:59]It enables right to left word handling to just work and all sorts of other features.

[03:56:06]See chapter 13 Latte Companion for other recommendations for language specific options or honestly just run search queries in Can. We heard a talk earlier today about expectations for uh like tags for search queries for citan. Um citan is really good. Like if you just type Hebrew, everything that is package specific to Hebrew will show up for example or Japanese or whatever your nonLatin alphabet language needs might be.

[03:56:36]Finally, a couple of uh oddities. First, I'm sure everyone here knows of microtype, but use package microtype I use in everything that I prepare because I love the elegant automatic things like improved automatic hyphenation, subtle justification changes and so on.

[03:56:54]Everything just looks right. It it makes it so that any document that a person is preparing looks like it was published by an academic press except it's just going to your own home printer, which is really quite impressive. Now for the online crowd, there's a package called Gregorio Tech that is continually updated and it enables the most powerful and elegant setting of Gregorian chant of which I know.

[03:57:22]Thank you to all the organizers for this wonderful conference. Now I will take your questions, comments, suggestions, snide remarks and more.

[03:57:31][applause] You like this? Okay, questions. Oh, right here.

[03:57:44]>> Hi, uh, Max Chunoff. So, my question is like we all know that latte isn't used that much in the humanities. What's the biggest barrier for that right now? Is it is it because it's too hard to use, the documentation is poor, or people just aren't aware of it? Like, what should we all be working on to make it more popular there? because that mean that's a goal, right?

[03:58:05]>> Yeah. Um, thank you for the question, Max. That's wonderful. Um, you've kind of put me on the spot because uh I don't want to say anything for the record that uh I might regret in some way. Um but but maybe the simplest way of thinking about this of course is um uh you know uh Steve Jobs is often credited with this idea that he foresaw that people would use kind of forever whatever computing tools it is that they were assigned to use when they were undergraduates. and and and uh we live in this word processor bubble for undergraduates that continues through graduate students that then continues through the entire profession uh say of a professional scholar in uh PhD studies in publications and so on and uh so I guess the great challenge then is uh breaking in way earlier than we might think we actually need to. Again, going back to that kind of jobs idea, right?

[03:59:12]Put the Mac in every school, put Latte in every school.

[03:59:15]>> Yeah. So, undergrad undergraduate outreach is you recommendation.

[03:59:18]>> Yeah. Yes, it is.

[03:59:19]>> Okay. Thanks.

[03:59:30]>> Uh thanks for a nice talk. uh you mentioned some a couple I think two uh documentation for the beginners in the humanity uh which is not so short uh blah blah and so have you ever consider learn laty which uh recently launched and would be beneficial even for the people from humanity science >> learn learn learn.org Yeah.

[03:59:57]>> Okay. Um, thanks for that. Uh, let's just add that to the record. Yes. Go to learnlawtech.org.

[04:00:04][laughter] >> Did you also use letterines or colored letterines in latte?

[04:00:15]>> Oh, I I have not, but um, you know, if I come across an instance in which I need to do it, I I definitely will.

[04:00:22]>> Okay. Yeah, I I know there are uh but the colored versions are a bit hard to set up. So I can provide a zip file or something with all the files in it.

[04:00:34]Thank you.

[04:00:38]>> Okay. So should we thank the speaker again?

[04:00:45][applause] >> Yeah. you know, in this environment where uh let's say we're we're not as fully conversant with the humanities as we might be. It's quite refreshing to hear a different point of view come in and give us some directions to go [clears throat] in. So, thanks so much for your talk. It was really good.

[04:01:08][applause] And so, now it's time for lunch. So, of course, I will say nothing else except we we meet. Yeah. Go for lunch. Come back 1:45. Enjoy it and we'll have a good afternoon talk.

[04:01:26]>> Okay. [laughter] Hi.

[04:56:23]All right, everybody. Everybody, lunch is over. Well, unless you brought a little extra back to help get through the afternoon.

[04:56:33]We [clears throat] have one very important announcement at the end of this session.

[04:56:39]If you read the fine print on the program, it's open mic time. So, there'll be an opportunity for you to uh vent or congratulate people or offer to host a future tug meeting.

[04:57:06]>> Okay, it's the last session of the last day.

[04:57:14]I'm reminded of a story that the physical society meetings on the last day the speakers would give their talk, pick up their suitcase and head for the airport. I hope no one is planning that today.

[04:57:30]Our first talk by River Temple and let me grab the introduction here.

[04:57:55]River Temple is an undergraduate at George Mason University, a biology student minoring in art and visual technology.

[04:58:10]Her professional focus for this presentation is on the intersection between design and science, asking what research needs to be pushed forward in spaces such as space travel.

[04:58:27]Typography is not always simply an aesthetic decision, but it is a tool. This will be her first presentation for conferences and the title from neurons to NASA a brief tour of the brain font psychology and implications in space flight missions.

[04:58:57]Hi, my name is River Temple. I'm a thirdyear biology student at George Mason University and I'm very honored to have the privilege of sharing my research with you all today. I really hope that my project can raise some questions and get you thinking about the role of typography outside of design because today I would like to talk about typography and space science. So the title of my presentation is from neurons to NASA a brief tour of the brain font psychology and implications in spaceflight missions.

[04:59:25]Let's talk a little bit about the brain.

[04:59:27]The brain is like a factory. It's responsible for pretty much most things that you do. And the way that it works is it contains a massive network of neurons that communicate with each other through electrical and chemical signaling. This is what enables you to remember things, move around, and understand what you're looking at and all the other good stuff. But in order to do this successfully, the brain has specialized compartmentalization.

[04:59:50]So each segment of the brain is responsible for unique things, but they all work together to complete tasks.

[04:59:54]Then there is rapid information processing and that is how your brain handles a steady feed of information. Um where it uses your memory centers and context to filter out unnecessary information and predict what is going to happen next and some of the regions involved in successful reading. Um this can be quite a large topic. So I've boiled it down to the things that you really need to know. Uh the first step is that your eyes capture information.

[05:00:20]That can be lights, rough shapes that gets sent to something called the fusifform gyus where that information is analyzed and depending on how it's interpreted different modules activate.

[05:00:30]So for example, if it looks like a face, there's something called the FFA or the fusifiform face area or if it looks like a character or a letter, uh the VWFA or the visual word form area gets activated. Uh which is more relevant with today's presentation. The jarus then takes all that information and sends it to your brain's memory centers like your hippo campus and the live processing feed gets rapidly inter interpreted based on everything you've seen or known in order to identify it.

[05:00:58]This all happens within a span of milliseconds all the time.

[05:01:04]So how does space travel impact the brain? Well, the first point that I'd like to look at is something called superimposed cognitive fatigue.

[05:01:11]Basically your neural feedback builds up and lags, right? Like a computer. If you have a lot of tasks going on, it's all going to be completed slower. Uh the second point is reduced cerebral profusion or cerebral hypoxia which lowers function further uh due to lack of oxygen. The third point is gravity induced loss of consciousness otherwise known as G-lock. Uh I'm sure a lot of us have seen those space movies where an astronaut is in an emergency and they have to get control of their vehicle and they start passing out. That is called G-Lock. um it is quite literally losing consciousness from gravity. And the last point is neuroplasticity, which is your brain physically rearranging itself uh especially more dramatically when it is exposed to different conditions for long periods at a time. And a lot of these might seem pretty negative, but they are really important. These are like your body's armor response to handling all these changes. So you really need these things.

[05:02:10]So delving a little further into neuroplasticity um some things that can happen are your cerebral spinal fluid volume can change the central sulcus of the brain can narrow and the actual matter of your brain itself can shift around and theoretically we can assume that this probably affects your reading and comprehension because the anatomy of your brain is physically changing. So here we have some images from the New England Journal of Medicine that showcase 94% of analyzed astronauts from long duration missions had their central sulcus narrow versus only 19% in short duration and it was usually less noticeable.

[05:02:48]So the period of time that you're exposed to these conditions does really matter. Talking a little bit more about white matter also there's something called the arcuate faciciculus which is a connection for the brain's language centers. You have your inferior longitudinal faciculus which is linking to visual processing. And these two work together. So the stronger that they're connected, the better that you can read.

[05:03:10]And dyslexia is what happens when you have bumps in between them. Right? You can think of it like there's a highway between the two. You start at one, the other is to your destination. If there's traffic, it takes you longer to get there. Pictured here is the Brocus area and the Wix area. The Broca's area is responsible for uh language interpretation and WEX area helps you with language output and understanding.

[05:03:34]So how exactly do you get reading out of this? Well, words are first broken down into something called phonemes. Phonemes are the combination of letters that form distinct sounds in words like the letter S and H creating the sound sh in that case would be a phone. These phonms are assembled to reconstruct word phenology in the left inferior frontal cortex. Um, but the letters themselves are actually converted into phonemes in the left supermarginal gyrus. Basically, all you need to know about this is it's birectional. So, it flows between the two to help you both understand language and produce spoken language yourself.

[05:04:12]Now, how are fonts registered by the brain?

[05:04:16]A lot of studies use something called ERPs to analyze how your brain responds to things. So ERPs are these little nodes that are placed around your head and they measure the voltage and polarity of your brain when you are reacting to stimuli. So in this case, it's written stimuli. It just shows how fast it takes you to do things and where they happen. And font disfluency is the subjective difficulty of understanding text within a particular font. Right? So a font that's in cursive and calligraphy is harder to read than a standard sans sarah font. Manipulating these fonts has actually been proven to slow down ERPs.

[05:04:54]So essentially we've gathered that fonts that look less like what we're used to in terms of characters do physically take us longer to read, which is important because soon we're going to be talking about how fonts and design and typography goes outside of just aesthetics.

[05:05:12]Uh so first let's delve a little bit more into font psychology. Fonts are very intentionally designed, right? Font design, it's an art form. One of the biggest driving forces behind art is emotional representation. And this is not any different from that. Think about if you were to see like a super geometric sans saf that's using a lot of angles in the character design. You're probably going to look at that and assume that it's technical, modern, and maybe even futuristic. And well, why is that? A lot of it is cultural association. Over time, it's led to our brand interpreting certain design choices with themes, feelings, and even places. And that's what makes company branding so successful. If you can get a customer to associate a font with your brand, you immediately have a memorable experience, right? That recognition really matters. And in the bottom right here, uh, we have the open dyslexic font. This font is designed to be more bottomheavy, right? so that characters that mirror each other are easier to discern. So if you look at lowercase B and lowerase D, you'll see that the lowercase D does have that extended line and even just small differences like that can help your brain interpret it faster if you have something like dyslexia.

[05:06:24]Now NASA's history with typography is pretty cool. The first example here is Futura. It is a sans saf font that was created during the bow house time. It has been used for a lot of different things, entertainment, uh political campaigns even um and it's associated with modernity and progression. It was coined the type of today and tomorrow and NASA has used it here in this example on the top left. This is a replica of the plaque left on the moon by the crew of the Apollo 17. This specific piece was transferred by NASA to the National Air and Space Museum in 1975. So, it's been there for a hot minute. It's really cool. The second image is the Challenger space shuttle which is using Helvetica. Helvetica being a sanser font named from where it was created in Switzerland in 1957. It's very balanced. Uh the lines going both directions are very even and clean, very legible, and it is used also for signage and things like that.

[05:07:26]The examples that I talk about here come from the brand guidelines. I believe these guidelines are specifically from 2024. Uh, however, some examples were harder to find on the actual website, but I'm going to go over them anyway.

[05:07:40]So, Inter, the first example here, was released in August of 2017 by Figma employee Raasmus Anderson. It was designed for computer interfaces initially, and now it's used for marketing and signage. It has this like tall X height, so it's really easy to read even at smaller sizes. Public Sands was based on the LE um it was based on Libra Franklin and this was actually developed by the government for government friendly uh branding by the US web design system team. This was released in 2019 and it's supposed to be neutral and clear. DM mono is a monospace sands uh safan foundry. It's for like code technical labeling and like numerical clarity. So you know all the like number stuff. Um, and this specific listing, like I said, is from 2024. Uh, which does also include some information on the legacy type faces like Heletica and Futura, but um, the three that I went over are the ones that are most commonly seen. And then you also have aerial present there.

[05:08:43]The information on this slide is really cool. It comes from a NASA display standard manual for uniform UI design across interfaces on spaceflight programs. So they want everything to look really clean and they want to have this like guideline for how you design these things, right? For usability. So some rules here are that the text has to be sans serif and you have to be able to tell what each character is super easily. It needs to be a fixed width font for numerical and tabular information just to keep things technically clean and modern. Uh the character height needs to be easily legible and colors are used symbolically to tell the crew members what the information means. So red is for highest alert. Yellow is secondary caution. Blue is advisory information. Gray is if your information is temporarily unavailable.

[05:09:28]And cyan is if it's completely missing or stale. And another interesting fact here is there's also a rule stipulating flashing text uh for immediate crew attention. It's reserved purely for that. And for standard frequency it's 0.8 hertz. And for super high priority it's three hertz. But the super high priority stuff needs to alternate between three hertz and 0.8 because if it's just super fast all the time, it's going to be really hard to read. It's it's quite an in-depth manual and it really makes you think about the usability factors of these UI interfaces.

[05:10:04]Some things to consider is that astronauts really need clarity on whatever they're reading, right? They have a lot to do and not a lot of time.

[05:10:14]And it's really serious stuff a lot of the time. You know, as fun as it is to go up there and just look at the planet from miles away, you're you're a very responsible person. So, um, font is really not just an aesthetic choice. It it becomes a tool in your arsenal. And when you consider how the brain changes in these conditions, whether it be stress or the time constraints or just, you know, even being in different gravity, it all slows down your neural feedback like we'd mentioned earlier. So you really need to think about the design choices that you're making here.

[05:10:48]Some future things that I'd like to explore in this topic. Um the main one would be if the open dyslexic font is beneficial or relevant for emergency UI on space missions. Right? If it was designed specifically for comprehension and discernability between characters in a situation where time really matters and you may not have a lot of it. The faster you can read something to get it done, the better. And I would also like to explore further on how these different gravitational conditions whether it be macro gravitational like centrifugation or microgravitational in space or when you return back to earth and you know that neuroplasticity your brain changes every step of the way. I'd like to look more truly into how that works. How does it affect reading and for how long do these effects last? And also, how could the world work together to curate fonts intended for spaceflight that represent larger populations? Could this be a discussion for different characters, you know, like Arabic or other script languages? And can we find a perfect solution for design and apply it internationally as a standard for legibility?

[05:11:55]I would like to thank Jennifer Claudio.

[05:11:57]Um, none of this would be possible without her. she saw something in me and gave me this wonderful opportunity to really delve head first into something that I ended up becoming very passionate about. And I'm super excited to uh see what else is in the works for this project. And I'd also like to say thank you to Steven West who is someone from my personal life. He was my soundboard for a lot of my research to make sure that I wasn't just saying a bunch of science jargon and it was actually, you know, legible and made sense to a large population. And sometimes I just need that. So that's been really awesome. Uh thank you for your time.

[05:13:07]interested in the cognitive aspects of perception and reading and uh I've noticed in your talk the same kind of trend that uh we we observed in the general literature which is that nowadays the most of the activity involving uh cog the cognitive aspects of reading are focused on on font shapes and not so much on on text as a whole.

[05:13:32]So I was wondering if you are also interested in uh studying the impact not of fonts themselves but on the arrangement of text on pages. For instance, what about uh interline dimensions uh whether justification is uh a good thing or whether a ragged uh disposition would be preferable for astronauts and stuff like that. So everything not involving the shape of characters themselves.

[05:14:01]Thank you.

[05:14:03]Um, I think that's a very interesting question and honestly I started kind of thinking about that myself during this presentation but you are completely right. A lot of the research on how font is read is definitely on font dfluency.

[05:14:18]Um, I do find it very interesting. I didn't get to delve into it too much given the focus of the content that I was finding and the research that was like pretty available to me for my topic. But that's definitely something that I'd like to look into more and I do think that that is an underrepresented aspect of the discussion.

[05:14:39]>> Okay. Now, >> thank you. Uh this is not from Okay, this is Boris Swissman. It's not from the chat. It's from me. As usual, I have a question and a very short remark. My question is this. uh you mentioned uh colors red, yellow and so on. Uh a lot of people have color blindness actually consider it and we have now standards about colors, luminosity and so on and I have an impression that uh the NASA guidelines have been written before this um research have been done or used. So do you have any comment or any um uh intention to go into the color choices by NASA and whether it's optimal and whether they should change it because a lot of people just don't see the colors like they think they >> I also thought about that myself because I I'm slightly color blind myself. Um, and you're right, it's definitely something that I don't think was considered when the manual was written.

[05:15:56]I think it might have to do possibly with eyesight requirement standards. Um, I didn't look too much into it yet, but I do believe that there are certain rules that you have to like have criteria wise for your vision to be able to go on these missions. And I definitely want to look further into any policies regarding regulation on color blindness. Um, but if there is more openness in that area during space travel, uh, I absolutely think that there needs to be more consideration for inclusivity within the colorblind community and changing even if it's just putting a little comment under or using symbols instead, something that could be understood by everyone. That's a really good question.

[05:16:40]>> Yes, thank you. Maybe we should talk offline about this and a very short remark. Great presentation. Thank you.

[05:16:48]And when you go to last slide, I understood that you are a student of Jennifer Claudio. Many people here uh know Jennifer. She was she had been in many TAC meetings and I just want to shoot out to her. She is she is always welcome and her students are always welcome in in this community.

[05:17:17]Yeah, Jennifer is Jennifer is awesome.

[05:17:19]Definitely shout out to her. I mean, none of this would have been possible if it wasn't for her reaching out to me and inviting me to this whole conference.

[05:17:31]>> And then like for everyone who doesn't know, Jennifer was the one who drew this logo on all their t-shirts and mugs and such. So that was her.

[05:17:41]>> Thank you. This is Carl. Thank you, River. It was a great presentation and and I wanted to just say there's a lot of other dyslexic attempts at font designs to uh help dyslexics nowadays. um and seeing a comparison between open dyslexic and you know some of the others could be interesting and I wondered for you are is this a area that you're going to be able to pursue uh in your academics going forward or are you on to other things?

[05:18:14]Um, I would love to I I think right now given that I'm still an undergraduate student, it's just completely going to be on who wants me in the master's program because I there's a lot of different things that I'd like to pursue, but this topic is definitely near and dear to me.

[05:18:36]Sorry for grabbing mic again, but it just happens that uh I'm a professor at the same university that u river is undergrad. So I need to think about this uh myself as well.

[05:18:54]>> I was going to ask whether or not you knew her.

[05:18:58]It's the first time I see you.

[05:19:02]Okay, if there are no more questions.

[05:19:10]>> Oh, >> all right.

[05:19:14]>> I actually have a actually have a short remark. Watch and learn, Boris. Oh, no.

[05:19:18]I'm just kidding. [laughter] Um, I am glad that I'm not the only undergrad student here. I'm very happy to see that. And also a very unique presentation. Thank you, River.

[05:19:32]>> Thank you.

[05:19:36]>> Thank Thank you to to everyone who took time out of their day to to hear me talk about this today. And it's super awesome that there's other gradu under other other undergraduates. I think this is a great opportunity for really everybody.

[05:19:56]>> Okay.

[05:19:58]All right, that's all. Let's thank RIVER [applause] and get on for our next presentation is >> and >> you need five minutes.

[05:20:20]>> Okay.

[05:20:24]>> Um I was I was going to tap dance but >> [laughter] >> The the floor is too soft.

[05:24:29]Okay, our second speaker in this last push to completion, Tristan Miller, is an assistant professor of computer science at the University of Manitoba.

[05:24:45]So, not as far away as Toronto, but [laughter] uh his research interests include natural language processing, computational linguistics, and all sorts of wonderful things.

[05:24:59]And he's going to tell us something very important, preparing those grant applications for those of you still active in academia.

[05:25:13]Very important.

[05:25:15]Okay, thank you very much.

[05:25:18]So, um, who here is in academia and has prepared grant applications?

[05:25:26][laughter] Okay, a lot of you, but I know I know from speaking to some of you, you know, we have some people here who are not in academia, maybe not really familiar with how the system works, uh, or maybe you're from a country that does things differently from how Canada does it. and I've only been here for a couple of years. Uh so um uh I I'm sort of getting to g grips with the um Canadian funding system myself. Um I'm going to be talking a bit about um you know one particular grant application uh for well for one particular kind of program.

[05:25:59]Okay. So um just a quick overview about research funding in Canada. Um it it's it's really dominated by the big three.

[05:26:07]They're called the tri agencies. uh and that is the natural sciences and engineering research council which funds you know science technology engineering type stuff. Uh we have the social sciences and humanities research council which funds all the social science and humanities stuff and then we have the Canadian institutes of health research which is all about health and medicine.

[05:26:30]Um and like I said it's these are basically the three biggest they're they're federal agencies they're the most prestigious. uh if you're a university researcher uh then you're sort of you know expected to get one of these uh a grant from one of these agencies uh if you want any you know hope of promotion or tenure or whatever um they've been around for a long time probably about as long as I've been alive uh and so uh but like I said I've only been working here for a couple of years uh so I don't have too much experience with it um people have written about the history of these um funding agencies. Apparently, back when they first began in the late 70s and early 80s, there was a sort of golden age where there was more funding available than there were applicants.

[05:27:21]And so, basically, if you submitted an application for those first few years, it was judged on its merits in isolation and not with respect to any other application that might be received. So, the funding decisions were always binary. you know, either you got the funding or you didn't. Um, and uh, I guess a lot of people liked it that way.

[05:27:42]But starting in uh, in 1982, the applicants, the number of applicants began outstripping the amount of available funding. So, they had to you move to a competitive model. Uh, and it's been like that ever since. So, now all these proposals are judged based on merit. Uh, and merit is is basically judged um, by comparing you to all the other proposals.

[05:28:03]uh and your proposals have basically um ballooned uh into these very lengthy complicated uh monstrosities. Um so uh like I said a lot has been written about it you know in in in blogs but also in academia people actually study research funding as a topic of research sort of I I don't know how they get funding for that but anyway so uh here's a quote from one study from Michelle McKinn uh who's actually in in this study is is it's sort of a self-reflection type study uh it's a co-authored paper where they talk about the experiences in applying for funding. So um they write the process of applying for funding uh is too much like an endurance test or a guessing game. It requires considerable time, commitment, teamwork, planning and sacrifice undertaken with the knowledge that the result may be nothing but frustration.

[05:28:58]Uh probably some of you can sympathize.

[05:29:00]I I imagine that this is not unique to Canadian federal funding. Okay. So I'm going to be talking in particular about insight grants. This is the flagship program from Sherk as you recall the which which funds all the uh social sciences humanities research. It's the flagship program. It's the most prestigious Sherk grant that you can get. You get the most money for it um provided you're successful. Uh so applicants tend to be not necessarily individuals though nothing stops you from applying as an individual. I think it's more common uh that the applicants would be small teams of researchers. uh and um all applications nowadays are submitted they have to be submitted through a web-based portal. Okay, they did away with paper applications already in the 90s. Um they moved to some sort of MS DOSbased system and now it's all on the web. Okay, so what does this application form involve? Uh well, it involves dozens and dozens of short answer form fields. Uh there's in fact uh uh lots of probably hundreds or thousands of short answer form fields you need to uh type your CV into uh which is lots of fun and can take several days. Um there's also uh between one and three long answer form fields.

[05:30:16]Basically, you know, you type an entire pages worth of text into it. Uh, I don't know exactly why because there's lots of other appendices on the form where they actually ask you to attach a PDF that can also be a page long or sometimes several pages long. Okay, so just to give you some idea of what it looks like. So the the insight grants uh u call for this year is now out. This is a a screenshot of the um um the first form in the application form. You can see here on the left, those are all the different categories of forms that you may need to fill in.

[05:30:53]Some of them are mandatory, some of them are optional depending on exactly what you're proposing. Uh this little white mark beside those categories, that indicates that a PDF attachment is required for those ones. Uh uh and possibly also some of these forms. So you see here uh a bunch of short answer forms where you actually type in a short answer or maybe here or here select things from a list. Okay. Uh then we have uh I'm zooming in here to one of the uh the categories that has this little symbol here. This is what it looks like when you need to attach a PDF. So you get uh this sort of uh instructions with specifications. As far as I know, almost all the specifications are usually all about the same. Um and then you just attach a file here. Uh and then finally we have like I said some some of these forms are actually long answer forms where uh you basically need to type a large amount of text or paste a large amount of text into this online form and there's you can see here there's a character limit.

[05:31:56]Okay. So the challenges in filling in one of these things besides the intellectual um um challenge of coming up with a good research idea. Oh this is my time is off. Um so applicants need to make sure that they adhere to all the content and structure and formatting requirements. Okay. Uh they need to also make sure that all of this content and formatting is consistent across all those different form fields. Again, some of which you type in directly into the web, some of which are PDFs. Um uh and so that means if you're writing it offline, as most people do, uh you need to make sure that um uh your proposal is consistent across all those different areas that you're going to be uploading individually or typing into the form individually. Uh because writing a proposal is, you know, a very long process, takes a lot of time. People usually invest months and months. You got to make sure that you keep all that stuff consistent across different revisions that you may be making. And of course, since you may be collaborating with a team instead of doing it on your own, you need to make sure that all your co-authors are on the same page and that you're all using the same formatting and style and uh that sort of thing. And of course, like I said, uh because most people are not entering their data directly into the web form, uh you need to take whatever workflow you're using, whether that's, you know, typing stuff into Microsoft Word or Latic as we will see uh and make sure that whatever you do uh is going to be, you know, submittable on the online system. So those are the main challenges here.

[05:33:29]Okay. Um so what I'm presenting here is uh something called Sherk Insight and this is a Latte package that I've uh developed specifically for inside grant proposals. Uh so the goals of this which basically match all the challenges on the previous slide. Number one, we want to automate the um the formatting especially of the PDFs uh so that they meet the sherk mandated specifications.

[05:33:55]um we want to ensure that uh they are consistent with each other uh in terms of style and in terms of things like citations and cross references. As we all know, Latte can only output one PDF at a time. Uh so you can't just put your entire proposal into one document and expect it to spit out, you know, 10 different PDFs all with the same citation numbering or can you? We'll find out. Um we also uh want of course you know in doing this of course you know I I like writing everything in Latte. So I want to support my own familiar Latte document processing workflows which involve things like you know writing in my text editor of choice uh using my versioning control system of choice that sort of thing. Um and of course uh even though I want to support all that I also want to make sure it's compatible with the online system. So no matter what I how I end up authoring the proposal, I need to make sure that it ends up spitting out all the data that I can directly import into that form without too much trouble. Okay. So what is the purpose of this talk right here?

[05:35:02]Well, um if there are happen to be any humanities researchers here um from Canada, well uh or who might want to submit to Canada to a Canadian funding agency.

[05:35:14]Well, I'm going to describe the functionality of this of this package.

[05:35:16]So it might be useful for you in that sense. Uh but um probably most of the people here are a bit comfortable doing at least a little bit of development work in Latex. So hopefully this will sort of serve as an inspiration to all of you uh to maybe build similar proposal templates and packages of your own for meeting you know whatever requirements of whatever funding agencies you normally apply to. Okay.

[05:35:40]And of course we can discuss my design as well. I mean I I make no uh promise that what I've come up with is the best possible solution. Uh and so if you see a better way of doing things, please let me know during the discussion period or send me an email or whatever. Okay. So this work isn't without president. A couple of years ago, I wrote a similar um funding uh like a latte package for Horizon Europe because I was uh in Europe until a couple of years ago. So Horizon Europe is uh the European Commission's basically largest and most prestigious research funding program. It funds you know work in the social sciences, humanities, medicine, I think um also natural sciences. It it basically does everything. It's huge. Um so it's got many different types of funding programs, individual ones. I wrote um a package for handling research and innovation actions or innovation actions. Um and so the the comparison between these two packages in terms of um what problems are they trying to solve um well for uh an insight grant you generally need to submit somewhere between seven and 12 PDFs to the online system but for a horizon year proposal it can vary according to the program but at least for the ones I've done you need only one PDF so this enforcing consistency across PDFs [clears throat and snorts] that's not really a big problem. Uh so we have a new problem here with inside grants.

[05:37:08]Uh the um inside grants tend to be you know much shorter than the uh horizon Europe ones. So maybe somewhere on the order of 50 to 100 pages whereas research and innovation actions you know they can bloom to many many hundreds of pages. So um that also sort of affects where you need to concentrate. Uh of course in both cases you need to follow all the funding agencies formatting and structural and um content requirements.

[05:37:33]you need to make sure that you know you're m maintaining consistency of the style and content and whatnot across revisions. Uh you know managing multiple co-authors um and of course uh because there's only one PDF for the Horizon Europe ones we don't really care about maintaining consistency across PDFs. Um the online forums for the Horizon Europe they don't have as many longans answer forums and the ones that do uh are really not so honorous. So we don't really care about maintaining consistencies between the PDF uh and the online form. Uh but uh one thing that you really need with the Horizon Europe uh proposals that you don't need with Sherk is uh calculating all kinds of tables. Uh basically this PDF this one PDF you do you need to put all the data for the proposal in it including you know the um uh the budget and the you know individual um collaborators need to be numbered and you got to refer to those numbers later in the proposal. You need to add up uh individual columns of the budget and whatnot. Uh so basically you need to have uh you either maintain a separate spreadsheet and laboriously copy all the information into your proposal and you redo that every time your budget changes uh or every time you add a new partner and need to do the remembering or you do it programmatically in Latte and let it do all the calculations for you. Okay, so that's why I basically why I uh I I wrote this Horizon Europe template is to eliminate all that problem. But we don't have this problem with insight grants uh fortunately.

[05:39:10]Okay. Um I'm not the first person to have tried to come up with some sort of uh latte or or automatable solution for inside grants um about uh 10 years ago uh a fellow named uh Christopher Bington Lei uh posted on his blog uh a Python script that he uses for sherk inside grants. Um but he doesn't use latte directly. He uses licks but licks can be exported to latte which is what he does.

[05:39:41]So this Python script basically um he he he writes the entire proposal in a single lick document. Then this Python script takes that licks document, converts it to Latte. Um it counts all of the characters uh in the long answer form fields and then you know makes sure that they're uh you know don't exceed that character count. it exports them to a text file that he can then copy and paste into the online form. Um the rest of the Latte proposal um uh he compiles into this one unified PDF and then he uses some sort of a PDF processing program. I don't remember which one. Maybe it's just Python itself. Um that splits that PDF file up into separate files that he can then upload uh into the correct sections of the form. Um, so that script is in in principle adaptable. Uh, maybe even to Latte. I'm not sure. It's not really designed for Lattic. It's designed for licks. Um, but he doesn't really provide any sort of document template. So there's no provision for managing the content or structure or formatting. You just got to figure that part out on your own. Um, um, the documentation is pretty sparse. It's basically 10 commented lines at the beginning of the Python script. um it assumes familiarity with Python. Uh which might not be true for every social sciences and humanities researcher. Uh I mean we can say the same thing about Latte, but at least with Latte there are some nice user friendly interfaces, right? Uh maybe not maybe not so much for Python. Uh that script also I mean it's it's obviously it's just sort of a a cluge that he he uses for for his own personal use. Uh he didn't really polish it up for for publication. So all the the file name paths and system calls and the page ranges and everything those are all hardcoded. Okay. And I and I don't think he's actually maintained this. I think that was just this one blog post 10 years ago and it's never been updated.

[05:41:36]Okay. So what am I doing? Okay. Well let's let's go through some of the technical challenges and the solutions um one at a time. Um so again another quote here. So team members had different style preferences, but coherence demanded a consistent style.

[05:41:52]Should we or should we not use serial comments? Uh, how about digital object object identifiers, DUIs? Uh, the principal investigator declared me the technical wizard because I could put asterisks in the margin next to sherk funded entries on our respective CVs.

[05:42:07]That's one of the requirements. Now, keep in mind that she's actually writing about presumably a word processorbased workflow, right? Not mine. Um, so the good news is that Sherk's document level formatting requirements are are are pretty simple. Uh, we saw them on one of those screenshots. It's just basically margins, paper size, and font. That's it. So that that's that's no problem. So Sherk Insight, you know, just in a few lines of code, and I'm not even going to show you what it is. You you guys all know the geometry package and whatnot. Uh, how to set fonts and things like font spec. Um, it automatically applies all that mandatory styling. Okay. Um, and the my package basically lets authors use whatever other styling rules they want because shirt gives you wide latitude to to format your proposal how you want uh subject to those u mandatory minimum styling requirements. And of course, if you're writing your proposal in Latte, uh, you know, this is based mostly on semantic markup, right? So, if you just stick to the regular, you know, section, subsection, uh, footnote, whatever macros, um, everything's going to look the same. uh it's it's a lot easier to um uh deviate from that uh when you're using a word processor, right? You got to sort of force yourself to use that style drop-own menu. And of course, we've all worked, you know, with uh co-authors maybe who weren't aware of that or forgot or or whatever. Okay, so it's much easier to enforce consistency that way. Um also with bibliography of course you know we have uh style files for bib or bibl um that you know we just enter all the information about a publication and it takes care of the formatting. So we don't need to decide collectively on whether or not to use DOIs right we just pick a bibliography style and the the program takes care of it. So this is nothing to do with sherk insight it's just a consequence of using you know the standard latte tools. Um, so Sherk doesn't care what referencing style you use for the references for the bibliography. That's fine. It just imposes two additional requirements.

[05:44:08]Number one, uh, any work that was funded by Sherk needs to have an asterisk in the margin. That's what we saw in that quote. Um, and student authors in any reference need to be annotated with a plus sign. Okay. Now, applying this formatting has befuddled a lot of grant writers. Um, again, we saw that in the quote, but it befuddled even me uh using a latte. I didn't I didn't know how to put a star in the margin next to a bibliography entry. Okay. But I look it up once I put it in a package and put it for every make it available for everybody else to download and use. Okay. Then nobody else needs to know, right?

[05:44:43]So I provide these ready-made solutions.

[05:44:46]Um provided the authors are using bibl.

[05:44:48]I do know how to do it in bibtech now, but bib is a lot harder to customize.

[05:44:53]You need to apply that to every individual style you use. Whereas biblate sort of divorces those kind of annotations from uh whatever style you might you might want to apply. So uh for annotating sh funded works um we just add a a keyword to the bibliography entry uh and then sherk inside package takes care of the rest. All we have to do then uh all it does is it it it changes this begin entry bib macro for bibl adds a box of zero width containing an asterisk. Um, so if you're a developer, simple solution. Uh, if you're a user, oo, it's magic.

[05:45:31]Um, so here's an example. So we just have, you know, a standard, you know, bite entry here. And the only weird thing about it is that it has this keyword here. And when we, um, when we process it through, uh, you know, beaver and bibl, uh, it ends up putting this nice star in the margin exactly like the specifications. Um, similarly, annotating student authors. I didn't really have to do anything myself. This is something that Biblate handles natively. Uh, and it's it's there for anyone who bothers to read the what 700page manual. Um, uh, there's something in in Biblate called a data annotated, sorry, data annotation. So what you can do is um add this uh field to the bibtech entry or a field in this format. Author plus am uh and then you just basically write uh the the uh the index of the author in the author list followed by equals and then you have some sort of special keyword and then you can you know change your class so that it applies some special formatting uh for any entries with that with that keyword. So that's exactly what I did.

[05:46:38]Um actually I don't think I've shown the uh uh the code here. Yeah. Okay. But uh this is basically how it works. So you have some other bite entry with a list of three authors. So you just add this extra field here saying author number one and author number three are both students. And then when you compile that document, oh there you go. There's the plus sign exactly per the spec.

[05:47:02]Okay. What other technical challenges do we have? Okay. Let's go back to Michelle McInn. What else does she complain about? She and her co-authors. Okay. Any change in one section had ripple effects across other sections and we had to check carefully to avoid any inconsistencies. When we added or removed research components or shifted items from one budget year to another, we had to ensure the proposal, the budget, the description, the student training, the knowledge mobilization plan, and all other sections were aligned.

[05:47:31]Okay? And keep in mind that you're doing this on a proposal that is split across multiple documents.

[05:47:38]Um Bington Lei uh wrote uh for the 2015 Sherk Insight grant there are two of the big text entry boxes and seven separate PDF sections to upload. One of those is a list of references. I would like to use the citation manager to make citations across the various PDF sections and to have the reference list coordinated across them. I would like to prepare everything in one file when I'm writing or sharing drafts for comments.

[05:48:06]Okay, so this brings us to solving basically the major problem here. How do we get multiple PDF files out of one document? Uh and um because if you recall, the proposal needs to be submitted as a combination of separate form fields and PDF attachments. But when you're writing a proposal, that's really annoying to work on multiple documents and keep everything consistent. Everyone wants to work on the proposal in one document, right? Um, so what I found was a package called subfiles. There's a package called subfiles that allows you to um, if you use it right, you can easily switch your compilation modes between uh, compiling individual documents or compiling one document that includes all of them. Uh, and the nice thing about that, it's not just, you know, including existing PDFs.

[05:48:57]It's actually handling things like making sure the the the citation numbering is consistent as we're going to see. So, how you have to apply this with with Sherk Insight is when you write your proposal, every proposal needs to have a main file and we could call it say proposal.te.

[05:49:16]That file needs to have document class sherk insight. Um and then it needs to include uh every subfile that is every PDF or whatever that you want to um upload separately or copy into the form using the subfile command. And if you compile that main file that gives you your complete proposal, you know, everything that you're about to submit.

[05:49:37]Each subfile is one of those separate documents. It needs to have the subfiles document class and you need to give it this optional argument which is the file name without the extension of your main file. Um now these subfiles will automatically inherit the main files preamble. So for example, you need to put only your styling information only in the main file. Your bibliography uh setup only in the main file. Okay? And if you compile any one of those subfiles, you get one of those individually submitted PDFs.

[05:50:11]So here's an example of what the proposal uh the main proposal file looks like. We do this. I've actually added a little um optional argument here because you can specify uh what year specification you are um aiming for because sherk might change the requirements from year to year. Uh you basically just you you can add your your um bibliography stuff here. Give a title to your proposal. Uh and then you just have a whole bunch of these subfiles each corresponding to what you saw on that that screenshot before. And then of course the the individual um subfiles, document class subfiles. And then you just write whatever you want for that one PDF. And notice here, even though this subfile is for the list of references, we don't need to include any of this biblex stuff because that was all in the main file, it gets automatically inherited.

[05:51:01]Okay. So, how do we ensure that the citation numbering is stable? No matter whether a subfile is compiled individually or or a part of the main file. Okay. Um, you can imagine that if your proposal has 100 citations, but maybe one given subfile of it cites only five of them, you don't want those five to be numbered 1, two, three, four, five in the subfile. You want it to use the numbering of of everything because you have only one file in the entire proposal that is supposed to list all of your references. Um, so the solution is basically to force all the subfiles to use the BBL file of the main file and sh insight handles that entirely transparently by basically conditionally loading the um, biblatic read BBL package. It's just a package that basically forces one document to use a different documents BBL file. Um, the only weird thing going on here that you need to watch out for is that you need to compile the main file before you compile the subfiles. Otherwise, they won't have bibliographies.

[05:52:04]Oh, I'm running out of time, aren't I?

[05:52:06]Okay. Uh, better better go quick. Okay.

[05:52:09]So, um, another problem is you might have broken hyperlinks in your in your document if you're just compiling one subfile. uh because it might you know you might you know link to some section or some bibliography entry that's in another file. Uh so the package just takes care of um automatically disabling hyperlinks if you happen to use hyperf when you're compiling a subfile.

[05:52:31]um for uh those foreign fields that you need to type in uh on your own uh rather than as PDFs, uh the um class provides a count characters macro where you can just type in in plain text as the argument whatever it is that you want to type into the form field. And of course the advantage though is that when you compile the main document, you actually get to see that uh you know it's it's not hidden in the web form only.

[05:52:58]Okay, so here's an example of that.

[05:53:00]count characters. Um, and that you basically just type in the um the text that you want to type into the form. It counts the number of characters and warns you also if you're over the limit.

[05:53:09]Uh, when it comes comes time to submit, you can just copy and paste that into the form.

[05:53:15]Okay. Um, I think I'll need to skip a little bit. Um, because these these slides basically just talk about the benefits of working with Latte and version control in general, which I'm I'm sure you're all familiar with. Um um McKinn was documenting an experience where you know she had four collaborators and was working on this proposal for three months presumably in a word processor and had to manage 206 different drafts uh you know from different collaborators and manually merge them. Uh but of course we don't need to worry about that when we're using something like git and latte or or overleaf or whatnot. It handles all that all that for us. Okay. Okay. So, just to um to conclude um you know, is what I've done a waste of time or or is this actually going to be useful? Because does anybody in the humanities or social sciences actually use Latte? But I I think there is good evidence uh and we've all seen it today um with uh at least with Corey's talk that yes, there are humanities researchers that uh that use um Latte. Um, I don't know if anybody knows, but uh a past president of TUG, Christina Teal, uh was uh a linguist and used uh tech uh extensively. She actually even authored a tech in the humanities uh column for for Tugboat uh for a while. I think it's it's no longer um written though. Um and so the argues that you know uh we tech developers or people who develop tech packages you know we should concentrate on reducing the learning curve for for non- tech users and make it user friendly for them. And I think we're at a point right now uh where you know we can we can greatly succeed. Uh it's just a question of you know let's let's write these packages. Let's get people interested in it. I've had some great experiences doing it. Uh I've submitted a few proposals using these packages.

[05:55:06]one of which was actually um accepted.

[05:55:09]So it actually works and I actually have collaborators in the uh humanities [applause] and social sciences. I'm I'm a computer scientist myself and they did not bulk at all uh at using you know technologies like like Overleaf and whatnot in order to help write the proposal. So I think it was a big success. So thank you very much. Um yeah.

[05:55:31][applause] All right, time for questions. I can see everyone's ready.

[05:55:42]>> Does it come with a gu guarantee of success in the competition?

[05:55:46][laughter] >> No, but I mean if you do get a something funded, then I I'll take a commission.

[05:55:53][laughter] Uh the question is a little bit more about the this subfiles package. Uh can it compile uh the the whole document and change the options or the the the options for the document class?

[05:56:26]>> Uh change the options for the document class.

[05:56:29]>> I I I'll give a concrete example. C can you for example just com my idea is to compile the same document with sometime different options for example draft and non-draft and doing it automatically without me keying in the the draft option.

[05:56:48]>> You might be able to do that. I'm not sure. You'd have to read the documentation for that one.

[05:56:57]>> Yeah. Both. Okay. Uh, are you planning on uh creating one of these packages for NSIRC proposals?

[05:57:04]>> I'll probably um write an NSIR proposal at at some point in the near future. So, probably yes, >> but uh let let me work on my sherk proposal first now that I have it.

[05:57:17]>> Yeah.

[05:57:23]>> Yeah. Frankie, uh just a quick question.

[05:57:26]Um the getting everything in one one file while working I fully agree with you that's very helpful but [gasps] why didn't you consider just sort of taking an offsite step and split the PDF in the end and and um make it that way.

[05:57:47]>> Uh well there were there there's a few hiccups if you do it that way. So, for example, that breaks the hyperlinks. If you >> Well, depends on how you do it, I guess.

[05:57:55]But, yeah, probably.

[05:57:57]>> Well, I don't think there's any there's no PDF splitter I know of that does that intelligently that will that will um preserve maybe. Well, yeah.

[05:58:07]>> Okay. Yeah. I was just interested why why why not? So that's that's a good argument if you have >> well another reason why not is you don't necessarily know in advance uh where to split the the PDFs >> because they might have varying number of pages.

[05:58:24]>> So remember the Python script that I talked about he hard-coded them because he couldn't figure them out programmatically but with subfiles >> you know you just need to know the name of the of the file. In fact you can put them all in a make file if you know or or whatever some sort of automated build system if you don't want to build them all separately.

[05:58:40]>> Yeah sure. things.

[05:58:52]>> Uh you mentioned that you now can basically write all of the content in your basically one document or split across a few files.

[05:59:01]>> Yeah. Could you then actually go one step further and just programmatically submit that through whatever API the web form uses >> programmatically submit this? Okay, I'm glad you brought that up because uh in in the like I said the the whole research funding and application process is itself the the objective scientific study and one of these studies uh makes the case that um the application process is complicated on purpose. It's basically put there as a a barrier uh so that they don't get so many applications. It's it's basically a filter. So they don't want it to be made easier. So um even if I were able to uh to do that, uh I think the the funding agency would see, okay, I've opened the floodgates. Uh we're going to change things uh so that it doesn't work anymore. Yeah.

[05:59:59]>> Thanks.

[06:00:14]this way. Uh since you mentioned uh these was for the shark proposals which are for uh humanities I believe and uh from the last presentations uh we heard that uh humanities aren't necessarily like the like latte isn't as popular in those areas. uh have you heard from users like common issues that they've had or things that would make it easier for them to like onboard onto Latte?

[06:00:35]>> Yeah, if you read the um the paper that I cite on this uh Haria class, I did sort of my own sort of post-mortem case study uh on the proposal that I I wrote with it with about 20 other co-licants, some of which were in this um social sciences and humanities, and I gathered comments from them and feedback. So, that's all summarized in there. Yeah, there we go.

[06:01:00]>> So, it's the this one right here. This disappeared in in Tugboat, right? Yeah.

[06:01:04]A couple years ago. And I I will submit a tugboat article on on this talk uh hopefully, you know, within a week or two. Yeah.

[06:01:20]>> Oh, which way is it? Is this way? Yeah.

[06:01:21]Okay. Hi. Yeah, Arash here. I think that's my name. Anyway, um so one thing um I I was curious about from like an accessibility perspective is um cuz I've been applying for a lot of these grants and I'm not in humanities um but yeah for the STEM uh STEM grants like um I was doing some for this research uh proposal that was accepted. wanted a few of them. And um so my question is when um the thing I I struggle with the most I think is yeah like the um they want certain things um u formatted a certain way and structured a certain way and this has to be bold and that has to be aligned and you check for this. Uh do you think from that like accessibility perspective because obviously I can't do it easily like I can do it in word but no no one wants to do that. Um and also I can't you know easily see I can't easily see at all period but also in this case I can't see the the rendered form. So working in in in like using your package do do you think that would kind of uh lower that that barrier a little bit from an accessibility perspective?

[06:02:28]>> Well I imagine it would because you're going to be using here only semantic markup and you're going to be entering all the information using a textbased interface. Uh so you don't really need to see what's on the screen. uh you just need to know the content of it and how many pages it renders to to make sure you're under the page limit. Uh SH Insight grants like I said they don't impose uh honorous uh formatting requirements at least not to the extent that for example Horizon Europe ones do.

[06:02:57]Uh so you're you're already sort of in the clear most of the way anyway.

[06:03:02]Sorry, it's a follow. So for academics, do you see this as specifically just for grants or other academic uh things as well? Because like you just mentioned that, you know, there aren't that many formatting constraints with with grants.

[06:03:16]But where I see them more in academia is writing things like um uh writing things like essays or applying for certain things and certain opportunities in academia. You got to write essays. do you think is something that you could use there?

[06:03:33]>> Uh, no. This is really built to solve one particular problem and you couldn't even use this for example for a different funding program from the same agency because the requirements are going to be, you know, too different.

[06:03:45]>> Yeah.

[06:03:47]>> All right. Thank you. Let's thank our speakers.

[06:03:50]>> Yeah.

[06:03:52][applause] And again, if anyone has any suggestions on how I can improve things, get in touch with me.