Python - Information Extraction Part 3 (2023 New)

[00:00:02]hi guys let's Now work on the third part of our series on information extraction specifically knowledge graphs so let's find that slide and get started perfect so the last video we ended on talking about how one can extract features and um specific relationship labels using snorkel and then potentially pull that as a way to build training data what we'll do now is pull all of the things from this lecture together and how to make a knowledge graph right so knowledge graphs are sets of structured pieces of information with their relationships and they're built sometimes into a network of information between a knowledge base and the knowledge graph and picture didn't show up at first but here it is so a graph might show you that Berlin is a city which is the capital of Germany which has the Brandenburg gate right versus France which has the Eiffel Tower so not only do we have the relational information capital of landmark of we also have the connection between these items okay and so the knowledge graph actually represents a large semantic Network and it doesn't have to be a network of places and entities it could be a network of Words which is the sort of research that I do but generally for information extraction it's going to be some sort of interesting information that you've extracted from your text and so for a long time this was called semantic web okay which was a set of knowledge that could be identified on the web and specifically these were things that had like persons places events that were built into notes nodes are the circles on these Network graphs and the edges the connection between nodes was how they were related to each other these two things are based on what's called an ontology and this is a more General description of how to make Network models okay so in semantics uh straight semantics what words mean this might represent having specific words that are connected to their features so birds have wings the have is the has a part of the ontology and they're connected to each other through that edge in a knowledge graph that might be like what we saw in the last slide uh is the capital of okay and so this ontology specifically that a lot of folks use for knowledge graphs is called a resource description framework or the rdf it's a large schema for describing things in their relationships if you want to talk about word semantics another version of this is wordnet that's very popular using text analysis and the statements are things like a subject a predicate a predicate and an object right Berlin capital of Germany this is a triplet of resources that are that are built into this framework that you can extract and then graph and so how do we create one of these graphs like how do we get from data to the graph unfortunately it's quite a process so we will do all of the tasks that we've talked about in this section so far we'll do named entity recognition co-reference uh resolution uh in the picture here it says conference resolution but I think they mean co-reference resolution entity linking relationship extraction and then graph the picture okay we'll also make co-occurrence graphs because those are really fruitful for understanding the relationships between items if you don't have a specific relation extraction capability so let's look at some example data and we're going to use the Reuters data set from nltk and look at Financial business data specifically where we're going to track mergers and Acquisitions and so we might look for in the data company one acquires company two company one merged with company two Etc and so that is ripe for rule-based extraction because if there's a specific language setup or structure with certain semantic key terms rule-based extraction is going to be easier because you just write some regular Expressions to extract that now I say that you just write some regular Expressions right and we've talked about how regular expressions are quite difficult and what you're going to see here is a lot of code to extract that kind of information from text but that is actually more efficient than training a machine learning model and you may not have the training data to start anyway so knowledge graphs are often fairly exploratory and we may start by using some of these rule-based extraction techniques and then move into machine learning once we have a large data set right so we're going to use the Reuters data set like I said and look specifically at their financial news section now let us see what we're what we're working with here so we're going to import nltk and if you haven't downloaded this data set you should I haven't commented out but just as a reminder that if you don't have the data um this won't work we're going to import that data set I was going to import pandas and read because this does involve regular expressions we're going to create a pandas data frame of the file IDs and we're specifically looking for the file IDs that deal with Acquisitions so it's basically using a subset feature to only grab that type of data because the rotors data is a very large news data set and it may have other stuff so we're specifically only going to grab acquisitions and that is based on this the way this data set's set up and then we're also going to load the raw text so what this does is it finds all of the ones that have Acquisitions in the file ID and then throws them into a a data frame so that we have our raw text stored in pandas now in this section we're splitting them up right and creating a data set that has the ID the um title of the or this is just raw right now so we have the ID or the row number and then the raw text so all this code is very specific to this particular data set you might have a data set that you just load yep so one issue is we always have to clean the data okay even pretty data has to be imported or restructured or done with somehow because it impacts the results that we might get so normally for information extraction we want sentence level structure so like normalish sentences okay what I mean by that is not just a lot of short phrases but sentence structure so in this example we're going to clean out the HTML we're going to remove the stock information that's specific to this data set replace the abbreviations that we know exist in this data set and drop the headlines fortunately the headlines are in caps so it does make it easier to find them and so we're splitting up the data um and so that now that we have the raw original raw text and then a column for headlines and a column for the text without the headline in caps then we have a very similar function to one that we used in the first lecture where we were cleaning up the text so my first lecture I mean the processing text lecture okay so we're going to replace all of the HTML um brackets their codes with the actual bracket part now unescape which is a function in one of our HTML packages sometimes works we had some issues with it in one of our classes but that would also work we're gonna sub out um quotation marks right we're going to drop stock symbols drop stop symbols in a slightly different way um we're going to remove specific combinations of of abbreviations so dollar DLR we're going to fill out to say dollar completely mln for million okay PCT um so we're also going to normalize all of the shorthand abbreviations into capitals so co-corp Inc PLC LTD notice how these are very specific to the question at hand take out quotation marks deal with white spaces uh fix Acquisitions being misspelled in the data set it fixed this typo in the data set so obviously at some point someone's looked at this data and figured out that there are only a couple of spelling errors fixed and fix all of those the cool thing is you can use this function not just apply but this function called progress map so it'll give you a little progress bar as it goes okay this works the same as apply so now our text column is the raw text cleaned up now what we want to do is find the things that only have uppercase letters so this is still a problem specific to this data what we're looking for so look here right the raw text that is only upper okay and print out the headline in the Raw text and so this basically says okay find everything that is upper and print it out so I can just look at it to make sure that I'm not losing anything I'm interested in and so what happens is sometimes the headline is the only thing in the text so we have the headline and then the headline again there's no actual text behind it because it's an old data set and so we're going to drop everything that is upper by using this not function so only hang on to the data that's not an uppercase okay in the Raw text column now we want to process this with species named entity recognition but what we're going to do is build a pipeline through space e that'll actually do all of the steps together and so what we are taking is we're leveraging A system that already has a lot of this information in it and adding to it and so I took a screenshot of this from the book but it has um a lot of the types of named entities that you might find in Spacey so person norp which is a funny one for political religious groups facilities organizations is mostly what we're interested in gpe Virginia political entity that's things like countries and cities versus locations like Mount Everest right products uh events so Hurricanes sports events work of art law and language so we're going to use these to extract the pieces that we're interested in first so import Spacey import display C we're going to talk about how to make some visualizations here load the small language model and at the moment we're just going to tell it to print the pipeline that is already embedded into Spacey okay we haven't done anything to this this is the pipeline that Spacey has naturally okay so toke to VEC is a token to vectors information it's kind of um kind of like word to vac not completely we won't use that one so taggers we've got to know the parts of speech right parsers this is the structure of the of the sentence attribute rulers right uh limitizers and Nar so we're going to start there and add new pipelines to Spacey foreign so entity ruler or the which is a different form of attribute ruler allows you to add entities to the ner pipeline okay it works a bit like regex but it's a little bit better than registrate regular Expressions because it's part of Spacey's neural net pipeline okay so it can learn as well and so we're going to add an example of government departments because we're interested in sort of the mergers and Acquisitions which is controlled by the SEC right and so from Spacey pipeline we're going to import entity ruler which is a pipeline that we can add to our model and we're going to look for justice and transportation we're going to add some more in a little bit but let's start here so um the Department of Justice right might handle some of these mergers and Acquisitions once they get themselves in trouble okay I'm not sure why the book has the example of the for Department of Transportation but we're going to add that one in so the hardest part about this is setting it up properly okay it is a dictionary so it's a list overall a list with a dictionary of dictionaries okay so let's look at this first one here because what we're adding is three entities to look for so the entity label will be gov to fit um to add a label to their system for gov we could do gpe or we could do um I don't know it's not a business so gov makes some sense so you could pick one of theirs already or add a new one um and so the pattern that we'll look for is text us and the op here indicates that it's optional and so we'll look for us um and that might be optional followed by Department of followed by one of these Justice or transportation okay and the entity type is an org okay so this basically says finally anything that says US Department of Justice U.S Department of Transportation or Department of Justice or Department of Transportation the US Parts optional it's we're filling in because they people tend to write it with this structure we could just have the department name first so U.S optional justice department which would be a little weird transportation department okay so the the words are being filled in here and then this one okay the end part makes it clear that there's more than one or a completely separate one Securities and Exchange Commission so notice that it's broken apart here by um brain fart by word okay because that's how Spacey processes things each of these will get labeled as gov the specific one does have an INT type associated with it so that means that normally this entity Justice and transportation will be classified as org but in this case we're going to label it as gov the easiest thing to do wrong here is to miss a closing curly bracket because it's a dictionary of dictionaries um or it's kind of a list of dictionaries of it's a list of dictionaries that have lists key pattern lists um and also to get the regular expression wrong but since it isn't actually using regular Expressions it's using more straight uh this sort of fancy Json format uh it's a little bit easier so you got to think about all the ways that this might appear in the text to write this well and so we've got U.S department of blank or Department of blank U.S blank department or um Department like Department department so to speak like fill in the blank departments or Securities and Exchange Commission now what about those abbreviations because a lot of people write doj as doj and the Department of Transportation as dot so we will expand those in a different component but here we're adding the expanded version as an entity so it will get captured now hi how we add the um entities to the pipeline and actually the book it suggests removing the pipeline and then at oops my bad removing the pipeline and then adding it back I don't know if that's necessary but I will say here's what the book tells you to do okay but either way you build an entity ruler so we play entity ruler NLP what's NLP again that's our Spacey model that we're adding to patterns equals patterns patterns was this page overwrite entities true so if they already had a name in there we're going to overwrite it with our new name so it comes up as gov um ruler add nlp.adpipe entity ruler so we're adding that in now this is a little unusual because the book has this wrong uh normally you just do NLP dot add pipe you'll see that in a minute but this one specifically requires that you do two steps it requires that you do add pipe and then add patterns otherwise you'll get an error that there are no patterns to be found okay so ruler.adpipe ruler dot add patterns and add the patterns to it even though patterns is here that was the only thing I get to do to where it actually worked properly okay so let's test this with some text and we can use display C to show off the entities found in the text which is a really pretty view of small amounts of text right so justice department the US Department of Justice dot security exam exchange but the sales department is not now notice that it's still found sales department and it labeled all of these as org so it's not totally perfect now how could we normalize them okay so we want to be like we've already done the text normalization but this is normalizing the entity names so we want to standardize the names to help with the linking between them and what happens sometimes is you'll notice here the Securities and Exchange Commission so in some cases it will pull in either a um uh possessive the apostrophe s or contraction and the word the okay and so I'm sorry to Ohio State University but the word the is not necessary so we'll remove that from the text so that we can get all of the mentions together and so here's an example Baker's International shares climbs the New York Stock Exchange if we process that with our Spacey pipeline before we fixed it print out the entities you'll see that the apostrophe s and the the gets stuck in there and that would limit us from combining them to the same object later so we can add what's called language components to space E3 so the book has both versions and a Spacey 2 and Spacey 3 but they're really going to move to Spacey 4 soon so we're going to do space E3 and so we're going to leverage their part of speech tagging to remove those patterns so to do that it's a little different you do at language component so this is where you're making your own pipeline so we can add pipelines that are already available in Spacey like entity ruler or we can make our own new pipelines okay and they're basically like little functions that you run on the data so it's kind of like an apply function and we're going to call this one Norm entities okay you've got to have this first line or it will not work then you just Define it as a function so for each document hold on to the entities okay we've got an empty place to save them and then for each entity and Doc dot ends because it's run through all the other Spacey pipelines first so you do have to put them in order here the pipelines need to be added otherwise this entity thing won't be there if the first entity is equal to a determinant okay so if entity 0 that's the first entity in the first entity listed okay so the first word in the entity is a determinant and the a then delete that bad boy okay so span says take the doc you take the entity from our document uh one word in to the end if the length of the entity is greater than zero okay so um this will capture um empty entities which does happen if the last entity is part okay so negative one means last if the last entity is a participle okay or a particle then do everything up to but the last one okay and that's what handles apostrophe SS and then return the document entities as a tuple so what we're gonna do is apply that to our pipeline so we've removed the first one and the last one so we just do nlp.i pipe Norm entities it will tell you hey you did this okay because we didn't save it as anything now let's print out the NLP on our previous text and you'll notice that we've gotten rid of the apostrophe s and the the at the beginning so now we've eliminated those from our examples and so the display C just allows me to show this to you a little bit more visually where we've the apostrophe s and the the are now pulled out of the entity recognition cool so we've added new entities and cleaned up them so they match now let's merge the ones that are all the same and so instead of having a bunch of tokens we want a phrase we want the entity to be one token instead of multiple parts of speech tokens um we just want the entity to be Baker International or New York Stock Exchange thank you and so we can do this based on Spacey's merge entities pipeline so notice there's no language component here because this is one they've already built so import emergencies uh remove it if you want I guess and then add it I think the removal is so that you get it in the right order I think that's why they have you remove it first but I'm not 100 because it just says this makes it easier in their notes but that would be my guess and we're doing these in a very specific order on purpose hey um and I think once you made a picture you would see the ones that weren't good the output wasn't good and you could go back and reprocess okay so we're adding merge entities and we're printing them out one more time right Baker International New York Stock Exchange so now is one entity as opposed to a list of of words that belong to one entity and so honestly this is like the paste function it adds them all together makes it in one one object now co-reference resolution okay and so this happens when you are trying to make all of the same objects be the same so now that we have them as one word How do we make sure that the all the different justice department options are listed together so if we look at this sentence here that they're using as an example in the book and then make a graph a knowledge graph of the options um or this is more of a co-occurrence graph how do we go from the one on the left to the one on the right and here The Wider the band the more they occur together and I will 100 tell you that I think um we're going to do this with Python's Network X but ggplot is so much better and GG graph is even better than that for these kinds of ones so while this is a python focused class if you can take the code uh the referencing information and Export it into R you would have some really great options for making really pretty graphs this might also be my R bias talking but I will tell you that I've been very frustrated with network X and that might be my inexperience so how do we make those come together and that uh task is called co-reference resolution and so here is an entire set of things that have to occur for that to happen Okay so we've got our instubular our norming of entity here and our merging and then this whole thing is the co-reference part entity linking we talked about um before this is where you would link an object to some known data knowledge database it's maybe not necessary for a Knowledge Graph and the only real free option is DB Spotlight I mean dbpedia Spotlight dbpedia Spotlight and you can add that pipeline but I will warn you their API has a certain limit to what you can do okay because I hit it trying to make these notes so we'll create an initial sort of mention cluster so initial co-referencing will resolve aliases like SEC becomes Securities and Exchange Commission well name resolve so we'll link together all of them that are the same and we'll do anaphora referencing okay now in this case that is actually done with neural co-ref which is a really great package that doesn't seem to work with space E3 so there are different ways that you could do this but I would say if you can do anaphoric resolution it would be cool this is things like pronouns it's there your ours are used to refer to it so that I can think of some ways that you can write this but it won't necessarily negatively impact your your graph too much um because it just will drop a number of co-occurrences because those words will get ignored and then we can normalize the names so let's start with linking names together okay and um what this does this is a specific um way to do this in Spacey Call initial co-referencing pulling everything together and so we're going to import token which is one of their functions and we're going to remove ref in and ref T from their function names first we're going to make two extensions so I've already imported it once I've just got it on here twice we're gonna set ref n and ref t and Define this function so for each entity in our document of entities if the entity label is the organization government or person okay we're looking very specifically for three these three entity labels then set um ref in as text and ref t as label okay so we're going to say uh if it's one of these three labels uh grab uh reference in the name and the reference text the label okay so all this is doing is just grabbing them it's like here's everything that's an org a government or a person with the text and the labels now this part's a little gross because it's kind of a lot of gross actually but how do I combine them all well you look at them and you can you manually combine them and there's nothing fancy here it's just a lot of code okay so we're going to display ner the display ner is actually a function that they've added on from the book to display uh named Etsy recognition for our custom system okay so essentially they've basically added the fact that we've got these um new pipelines in here and it should display based on the new pipelines and not just the old ner pipeline so that's what this one does uh this one makes a pretty difference tea tree dependency tree right neither of those are necessary they're just useful functions so let's look at what actually is doing the work here how do I make sure that everything comes out and looks the same and that is just a find and replace so here are all of the acronyms that were in those entity lists so effectively we found the ones that or the organizations in the government and we printed them all out basically uh you could make a counter a table of them especially now that our class has figured out how to do this so we take all these entities we print out a frequency table of them sorted by name and then you come up with this list oh okay AMC is American Motors Corp and so you just essentially are are fixing all the typos it's another great place to fix typos you're correcting all of the uh shorthand abbreviations to the long versions and this would require you understand what the data is obviously so you make this acronyms list and we say um Alias lookup so I don't know why SCC is not just at the top here I think it probably just got added later like I think these were from um a previous chapters example and so we're adding our new acronyms so um we're creating the acronym list here so this is a dictionary of key value pairs then within that every one of these is an organization okay so we're adding a entity text to it so basically saying like all of these entities should be labeled as Orcs for sec we're adding it in as a a tuple okay of of the expanded definition for SEC so security is an Exchange Commission and it's a government entity so essentially this becomes AMC is American Motors Corp and org Ami American Medical International Inc and org so this is just a shorthand way to add the same label to all of them then we're adding a couple more here so uh the U.S Department of Justice is a government and it it can be listed as any particular one of these three U.S department of justice department Justice U.S justice department and Justice Department so this is a shorthand way to add a large set of key value pairs so it has one key or it has the key value key Pairs and then all the ways that it could be listed you could also do this one at a time up here so we're adding Department of Justice Department of Transportation this organization USA air group and Transworld Airlines and so this is sort of three different ways to handle the same problem the problem is we need a key to a value and an entity well these These are the only real issues with them so expand these acronyms out then uh also add sec but last so to speak we have some that have multiple potential listings so this is the final key value pair that I want um I'm sorry not key value pair this is the final Tuple I want all of them to say U.S Department of Justice comma gov and these are all the potential keys that go with it so if you see in the document the key is what might be in the document make it this value if you see dot make it this value okay so there's no particular reason to have like three different ones except they're showing you different ways here to handle a bunch of of of uh acronyms versus a bunch of keys to to one value so then we update our lookup table and so the lookup table is simply finding that piece in the text and converting it and so this is an example of what it will do once you use the lookup table okay so for transportation department it converts that into U.S Department of Transportation which is government for SEC converts that to hear Securities in exchange and that's government so this is what it's going to do you know how do I actually apply that in our pipeline so we'll write a little function called alias resolver okay and it says okay for all the entities what is the first entities token name okay and so if the entity token name is doj if that token is in the Alias lookup well doj is in the lookup then grab the name and the type and add it to our new name type um placeholder okay and propagate entity type just puts it back into the the pipeline and so basically um propagate entity type is actually defined here sorry but what it's doing is putting it back in the pipeline so for all the entities in the document entities basically um for the entity we're currently working on um figure out where it starts and ends and add it back into that Tuple pipeline so let's look here the deal of trans World Airlines which is one we're going to convert is under investigation by the U.S Department of Transportation the transportation department will block the deal of TWA and what that's going to do is display so we're using our display function at the moment it's not fixing them but it does print out that they are um both government agencies right and Orcs And what this will do once we apply this pipeline is it will transform them into all the same word all right now name variations sometimes are when things happen the first time is listed as their whole name and then maybe the next time is listed as part of their name and this is most commonly with commonly done with human names right so if I said Dr Buchanan blah blah blah blah blah blah then later it's just Buchanan blah blah I found this very awkward when people write this way because I solve a first name right um but this is the way that a lot of articles are written right and so what this does is it will um look up and convert them all to the same thing okay now one problem that we're having here in our our output um is that we're not seeing the whole table so it doesn't look like it's changing I said a minute ago it didn't change it but what we're actually seeing is that the we're printing the original text The Entity type and then the new reference and reference to the new stuff uh and those are actually corrected you just can't see it all in the not awesome printout format okay so when you're looking at this I'm running this one line at a time in markdown for example which is how I did this you'll see that it actually does convert them properly and we'll see that in our final picture as well all right so find okay so sorry first resub ignore parentheses and dots because um abbreviations find the word boundaries right so remember that slash B here means word boundaries so don't pull an abbreviation from the middle of the word because SEC is the first part of secretary so don't screw that up okay uh clear out all the spaces right so no Extra Spaces squeeze them together and um return the object as long as it's not none okay so once I clear out the spaces and mark the word boundaries if it's essentially no word there just move on okay and so our name resolver um says okay find all the entities that are listed as org and person okay if the first um if the name for the first reference to the entity matches the second text then rewrite them and so here Transworld Airlines and TWA are both converted into Transworld Airlines um partially because we've already did that with the previous um acronym resolver uh but here this will resolve first and last names and so essentially we're trying to see if if the like last name of the first one matches the last name of the second one okay and it does work if we saw all four columns now strip all the legal suffixes this could be Mr Dr Esquire Junior if you're doing people's names but in this case we're doing uh business names so we're just going to take all of those off and this is a simple regular expression where it just says you know if you see any of this stuff delete it so if we do strip legal suffix suffix it turns out without the code now we're going to call that norming names so if the reference name isn't blank and it's an organization then strip the legal suffix and return it and we've added that pipeline in Okay so we've gotten through all these different pipes how would we figure out how to do these you would make this graph it would be hideous and then we go through and find ways to prune you'd say oh I need to take all of these acronyms out of here and convert them all to the same oh sometimes people just say Huges and I need to make sure that that's resolved where this it's the same name and so my recommendation actually is to get through most of the um initial co-reference resolution and then just print it print out a table of them hopefully there's not Millions and especially uh you know you don't want to make these graphs with thousands of of um points anyway you won't be able to read it and just go through them and say okay this is one that this one and that one or the same thing and so we need to we need to fix that how do we fix it well I can use a regular expression or I could say expand This Acronym um so once we have finally have all the data together we let's start with making a co-work excuse me a co-occurrence graph I personally think co-occurrence graphs are like the coolest things in sliced bread but that's because this is what my research I use to visualize stuff but essentially what co-occurrence graph is is exactly how it sounds it displays entities in this case that co-occur together in the same document and one of the most famous phrases in word nerd land is if uh words that are related are in the same documents okay so if they hang out together they're probably friends and so they're related in some fashion we could say that they're semantically related meaning they have similar meanings or in this case they're related to some something thematic so uh they're all businesses because they're in a document about businesses and so we can start to map relationships because it just happen to occur together in the same document now document we can Define as the entire paragraph We Could Define document as one sentence we could Define document as like a whole news article so that level is dependent on your own interest a lot of people use sentence level documentation but for some of these more complex um if we're using entities we might consider like paragraph or or whole document level um co-occurrence and so we may not know the exact relationship from these things but having them linked together and having a picture of it gives us a big clue and this is a pretty graph version that they made with uh gelfius and I think how you say it but I can't get their output from Network X to work with it currently and if I look on their GitHub page it's a currently like to-do list item so we'll look at the graph from Network X and then I will tell you that if you go look at tidy text mining the book there's a really great example of making a network plot with GD graph and it's really pretty so this is what one might look like okay or each dot is a node with a name and the strength of the connections is wider for more connections and this gives you an idea of the communities in the text so everything here is tied to the SEC but we have like oil and petroleum we have TV music Communications Aviation City Core Audio Video okay so it's kind of interesting that they aren't paired with TV down here Reuters all right we've got cars and this must be Financial stuff and so this is a really great exploratory analysis for understanding the underlying um kind of Connections in the text so let's look at how to how to make that okay so how do we do this well we're going to use our new fancy NLP pipe to run this and I will tell you this is very slow because not only are we piping everything we're also grabbing those co-occurrences out of text so pop your sleeves and prepare to wait because the more texts that you have the longer this takes so we'll say this document because there's two of these in there it takes like somewhere like about eight minutes to knit the slides and it's all because of these two slides that's not terrible for 2 000 documents that's going to be better on better computers but if you have millions of documents you might rethink some of this okay all right so we're going to import math to get the ceiling and TQ DM now don't do the stupid that I did that took me some stack overflowing to figure out but tqdm is a function in tqdm the package because that's not confusing at all so be sure you say from tqdm import tqdm or you're going to get a weird error message that thankfully someone has answered on stack overflow okay so let's make ourselves a function extract co-occurrences from the document and the include the type of thing it is okay so our entities our list of entities is the name reference and the label remember that's the entity so name reference and label is the original name and the entity type for every entity in that process document and include types so a clue types here is where you can set what kind of entities you want to pull out okay now the set function here means don't repeat them so this is the list of all of the entities and their combination so if U.S Department of Justice is listed a gov wants an org a different time it will actually come up with two different ones but we've handled that previously so create a unique set of entities and their names okay which is why you gotta normalize them first and um the yield from make the combination so count basically now the batch size here you can set it to whatever you like I did it as a hundred so it'll do 100 documents at once and the number of batches is based on the length of my data frame and for intqdm range zero to length right based on batch size and run a total number of batches so this allows us to like process this in chunks so it runs faster run the pipeline okay on our text column our text column is is our cleaned up one that we did the basic level cleaning and our pipeline is like everything that we've told it to do on I to batch size and I here is going to increase based on batch size so basically this says run from zero to 20 900 I think rows based on batch size so 0 100 200 300 400 okay so I is increasing in units of 100 because that's what we did up to the total number of batches grab those first hundred documents and try to um count the entities that are either organized organizations or governments so this line here adds the um the co-occurrence output to a data frame okay well it's a list actually if it doesn't work print the index and print that date part of the data frame and blow up an error so try to run it and if it don't run here's an error message so that you can figure out why it's not running now the cool thing about this package is it's going to write me out some output here so this tells me you know how long it took how long it ran so this is how long each batch took so that's why it takes about eight minutes because you'll notice here that some of these batches took almost a whole minute all by themselves and now just to print some of our co-occurrences okay so the first one is here there are 545 co-occurrences of American International which is an org and the Security and Exchange commissions which is a gov there's 1735 trans World Airlines and Dot and U.S Air Group and so now we've got a tuple that has the number of co-occurrences the first occurrence type which is the name and the um name and the um little entity the second occurrent type name and entity and if I convert that into r i could treat this as um a data frame and use the GG graph option to calculate based on frequency and I could color them for example by the type of entity I just really like GTO graph but let's see how we do that in Python and so we're converting this into a pandas data frame right so we've got their ID entity one entity 2 for each of these okay so the ID is the number of times something occurred entity one is a tuple into two is a tuple and make some columns we've got the article ID The Entity type in one type one and two type two so this converts it nice into a pretty format instead of the structured list of tuples now this functionality I think is really fun mostly because I have been playing with spark X Sparks spark Apache spark Pi Spark so this kind of actually pulls in and runs this a little bit to me it looks like Pi spark and I didn't realize that I could do this with pandas in regular python so that's kind of fun but um I do wish they hadn't completely switched uh types here but Group by okay Group by groups it by the two um the pairs of entities so group I essentially just like d plier groups them together where what we see is that like this set do something on this set and if the other set is slightly different do it on that set so kind of keeps together the pairs and their types so it'll calculate this once for all pairs and types okay AG okay AG here means to count we're creating a count and a sum and so what you see is you end up with um the the frequency column of the number of types I lie to you this is the article number I lied I forgot this this is our in this first article article number 545 we have um these two listed together in article 1735 we have these two listed together that's my fault sorry so now we have um a data frame of what of um the article numbers and we want to sum them up that's my bad so AG here means count okay count how many times this combination occurred now sum here is a little weird because what it actually does is since these are text components it just like creates a list of all the articles in which they're paired together we're going to rename the count column into frequency and the sum column into articles so it's more readable and reset index on pandas here will just make it um a list of um like the row names will be one two three four five and let's see articles oh okay this joins them together with a comma so this is what output we end up with and these are the Articles we could go pull to look at how they occurred together if we're interested with and so let's see what that looks like now that we have it in the right format to make a graph and I will tell you I'm not that great at Network X at all but here's how you do this in Python so we import Network X and you must remember to import matplotlib because it uses matpodlib in the background if you don't none of the code at the bottom will work and you'll be sad so NX Dot from pandas cool we tell it what columns we want entity one entity two uh the article numbers which I don't know where these get Incorporated and the frequency we're renaming frequency to weight because it likes to have the column name for the strength of the pathways to be weight okay Source entity Target entity 2.

[00:55:15]so who how are we connecting them we're connecting one to two okay based on this weight um Edge attributes true now I turn this part off because it made me a little triangle but if your document is too busy there's too many things on it you can tell it to graph only specific components okay I turned that off so this is just one of the many layouts that's available in Network X this is how big to make the picture now draw me the picture so we're going to draw the graph and the position okay POS here stands for position node size 1000 node color sky blue alpha 0.08 alpha is like how um transparent it is with labels equals true so it puts the words on there there are so many things you could do to make this prettier okay as long as you can get it into a graph you can use it visually but if you're wanting to put this on like your website definitely spend some time looking at network access documentation to make it better looking now we've drawn all the dots now we have to connect the dots so for each node 102 and their data graph the edges okay the width is based on the weight from above draw Network edges graph part of speech Edge list node one node two no one no two or entity one and cc2 width equals the width my Edge color here is black well light gray and Alpha is 0.05.5 okay PLT dot show so show the picture and this is what we end up with now what could we do to make this better one we can make the font size larger okay um and I tried to close the graph so it would stop giving me this um cute little thing but this is a problem with um Corto documents I think so just kind of ignore the white boxes that won't go away after this one but uh what we see here is um where's dot Department of Transportation is only one word Department of Justice only one word so proof that all those documents later it's only one node instead of multiples okay and what will we get will we get a little a little group of folks up here okay uh and the Securities and Exchange Commission so we've got this Bank the presidential commission on good government San Miguel which I don't totally get that might be a label that's just not correct because it doesn't appear to be an org or business but I could be wrong that's a place I thought um and then what we see is this like three-part system of Transworld Airlines U.S Air Group and Dot so the I'm sorry yeah dot so there's a lot of action going on here in those articles so that might be the time period of something going on which is kind of connected to another airline American Motors right the Interstate Commerce Commission which isn't too surprising it's not too far away from Motors I do think it's interesting that those go through Department of Justice and not Department of Transportation but okay the FTC right and so we can kind of start to see some little families occurring here but the biggest three connections are definitely these and that's what this graph giant component will do it will pull out and only graph just this one don't do that you'll lose some information do it later after you've looked at the main picture the other thing you can do is [Music] um in your pandas data frame exclude a lot of the low level frequencies so if you had a ton of data you could say okay only give it to me when their co-occurrence frequency is over 10 for example now the last component here is for the actual Knowledge Graph and I would say this doesn't work very well in this data set but I was kind of like 87 slides in so too late but uh what we could do is pattern based relation extraction okay based on the sentence structure this would be arguably the it's not easy because you've got to figure out what the pattern should be but it does require sort of less data to get started and you'd modify that information based on the topic at hand so if you're talking about spouses remember with snorkel we did like married uh wife husband right if we're talking about mergers and Acquisitions we're going to use merge acquire and that kind of stuff and so we're going to use acquire sales subsidiary subsidiary of or chairperson of okay and one of the issue with this analysis is the ambiguity in which we write things so you know we don't always write using the word acquires or merges and so uh the sentence level structure may be different across these different examples so we kind of have to figure out how to deal with the fact that things are written in different ways but mean the same thing okay and so we'll build this giant pattern matcher so buckle up right so we've got spacey.matcher we're importing matcher and we're going to grab all of the vocabulary that happened in our NLP when we ran NLP so here's the vocabulary list of things that are available what that does is it gives us um the all the different options so I think this would would handle um acquire it handles acquires we could do buy bot purchase purchased so we could capture all of the the different uh potential combinations and we could uh um really focus on the lemmas here so we'll talk about that in a second but we're gonna do the three root lemmas acquire buy and purchase so bot being the past tense um whoever the subject is okay so this is empty meaning whatever is listed there as the the um with this subject who is an organization followed by um a reference not in the organization but that's optional followed by our verb where the limma is in these three so that controls the past tense of purchasing purchased purchases acquires is apparently a weird one because acquire and doesn't acquires doesn't seem to map back onto acquire singular so Spacey's limit detection is not perfect so this just gets both of them but by using the word limit here uh what we're saying is like as long as it's the original word is by that's fine okay so any particular subject name it's an organization potentially followed by something that's not an organization but it's optional remember op here for optional followed by um one of our words so we could say General Motors acquires something right followed by something not in an organization again optional followed by org okay so effectively we need to say something like General Motors acquires Apple I don't know that wouldn't work but you know that order and it will also pull and allow things to be in between okay so like the chairperson's name or something else and this was determined by looking at those patterns now here's where we can get subsidy of or unit of right so subject's name not uh something followed by not an organization where the part of speech is not in the verb but it's in this the uh synonyms of the verb so this handles subsidy of unit of something not in the organization part of speech uh not in our verbs it is optional so this one will handle General Motors a subsidy of Chrysler let's say a little way around right or whatever it is okay so these look for those particular patterns and this function here deals with the patterns so here are the patterns extract relationship match now for each sentence for each match ID start and end in the sentence find how long the item is uh look at the rule names the subject and the object of the sentence are the first one and the last one and this one will handle the reverse so if the items are reversed still keep them connected right so if I say uh if I said my other one the other way Chrysler subsidy of General Motors with the General Motors the subsidy of Chrysler like that would keep them counting as the same thing because they're just reversed I think and yield What's It Gonna What's It Gonna return is the reference in reference T the relationship between them and then the reference in reference t for the object so the first object it's prediction so it's pattern subsidy of uh acquires okay and then the second one and one more pattern um reverse the reverse subsidy of okay now um I don't know why this it prints but it doesn't show the it shows this big white box this is like a weird bug going on but essentially Foods Fujitsu I can't say this word plans to acquire eighty percent of Fairchild Corp okay so it would grab the first word here acquire and Fairchild Corp which is an industrial unit of Schlumberger hey so we'll grab Fairchild Corp unit of slumberger okay and so uh it would extract those and then here you could if you run this code it'll print out for you right now acquires is a weird one so if we look at the output when you see the output acquires doesn't work that great so we added it in to our pattern match here for choirs and ran it and went oh if okay because people use acquires in a lot of ways so we're actually removing it that's not necessary you would look at the output from some of the text and go oh that didn't work right so instead of using pattern matching okay we could instead use dependency matching now dependency matching is a bit more complicated which is why the code longer but what it does is it grabs the dependency parsing structure okay so dependency parsing structure is where we have um a verb as our mainly kind of root word and from that we get the subject of the verb and the object of the verb and we're looking for acquires for example that's going to have a subject and an object so we can pull out the sentence find this specific verb and grab us dependent and subject and it's dependent in object okay and that's what this does so it walks through and says um If the child dependency is one of the entity types we're looking for now the child dependency is if the root word is the verb it has a child and the child is the object or the subject and so if if the child is one of our entities return the child else do something else else if the child dependency is a compound now this is for compound phrases like Fairchild Corp is actually two separate tokens right now and um they're listed as compound words so it just allows you to grab both parts of the compound check it and so if it's part of that return the child this part deals with passive voice okay so finding when you say uh is acquired by so this allows it to handle passive voice find the closest subject in the predicates of the left or the parents left right so this um fine subject uh takes the passive voice option and finds the subjects based on the passive voice thank you and find object here finds the closest object when um in the regular voice so the object is the second word I kicked the ball kick here is our word I am the subject the ball is the object so that now we have the connection between the two acquired subject and object um if you said the ball was kicked by me oh so awkward uh we've still got the same pattern but wow we're dealing with the passive voice where there's an auxiliary word in there was kicked and so that's what all this does so let's put all that together so this is everything so we've got our patterns from before okay and so where we took out acquires because we're going to let acquires be handled by the dependency tree because we figured out that the dependency the pattern matching doesn't work very well for acquires so find subsidiary fine unit add all these things to it find chairman president director CEO and executive so we're going to pull out the people involved in these relationships too and we're going to extract relationships from our documents and so we'll extract the relationship match we'll extract the relationship dependency using acquires and to extract the relationship document dependency using cells so the matches occur from right up here based on these words and then we'll also use those two verbs now we run this code I really don't understand this picture thing but uh what it does is it prints out are pretty graphed out of the bottom where it tells us how far we've gone okay so we'll run 20 at a time and for the full number batch size we're going to run the same thing here pipe it through um and here we're just going to extract the relationship so relationships.extend is basically adding here are all the relationships so we're going to end up with a list of lists of the um you know return all the stuff from extract relationships so we'll have item number one the relationship and then item number two we'll put this into a data frame of article subject subject type predicate object and object type I'm saying that predicate relationship um you can't be both the subject you can't acquire yourself okay so take that out um this is just an issue with the tokenizer so take out things that start with own drop the duplicates and aggregate them together okay so this code very similar to the previous code now I will print out some value counts and in just a second I'll go back to the markdown and we'll look at some of the outputs for these and make the graph now this graph is not nearly as great because um it won't print the relationships between objects in um the multi-diagraph mode and I'm not sure if I turn this off it will but the multi-digraph is supposed to allow you to export this to a separate program called gelfy but it's not working because the code is outdated so I just told it to print [Music] um straight from the visualization here but now we can start to see how these are directly connected through the text as opposed to co-occurrence right um and so we'll pull up the chart here that that produces in just a second and look at what is actually in the data frame but at the moment because we've turned on multi-digraph um we can't get the relationship to print on there but there was a reason why you needed it all right so let me wrap this up I'm gonna get my markdown started actually because it's quite slow and we might have it already although it does um it does tend to give you yeah so it's already pulled up Grace it does tend to lock up sometimes because of the displace the output and I want to tell you one more thing about the splicity before we go but let's wrap up all three weeks oh my gosh we've done so much code okay and when we'll do together in class is see if we can modify this code on new data because that's hard part right um is thinking about okay well how can I apply this to a completely different fields and tough from different topics it requires a lot of thinking okay there is a lot of digging around when you're working on um extracting information because of the flexibility in language okay so what we covered was key phrase extraction which required some part of speech tagging named entity recognition and linking those named entities to a dictionary database relationship extraction here at the end um and how to put all that together into sort of a final output of a knowledge graph okay so let's go back to markdown real quick and look at some of the outputs that were sort of weirdly blocked by my um nonsense okay so as we're removing things and we're dropping duplicates this is the chart that we end up with right oh geez and it's still not great I need to figure out if I can make this Jupiter do I have objects is this fake run I think so I don't have it saved but we do end up with what articles they're in and the frequency of them occurring together right and then we're having it print out the relationships so let me hit this button so I can run everything and I will warn you it is a little slow it also prints much better when you're working interactively and so we'll look at some of these graphs or not graphs some of these outputs that we couldn't see before and we'll turn off this silly output so this is one of the many things that annoys me about uh markdown sometimes is this output but what is that crazy output well the problem there is um displacy so all this is fine yada yada yada still running as you can see at the bottom it's still thinking so this output's fine where's that displacy let's find it okay so display C is really cool but no matter what you do in that I've seen so far if you're using Jupiter it shows up fine if you're using our markdown here or quarto like I'm using it shows up like this okay this is an SVG it's building a HTML page for you that's cool cool story bro but it won't show in the output it just shows you all this crap okay and what I've found in markdown and in quarto anytime you use displacy you need to make it a separate chunk so don't put it in with another chunk and make sure you have results equals as is okay that will force it in HTML format to knit to HTML and actually show you the picture as opposed to just showing you a bunch of junk okay so make sure you use the results equals as is we're almost there it's running down here at the bottom and so let's see close but then it it does print these giant giant screens sometimes all right so graph relationship matching putting it all together very cool okay so we had it print some statistics down here at the bottom but let's just look at it so you can use the view function when you're working interactively and that works great but man telling it to print it just like dots The Columns so tidyverse is something right there but let's see the subject of the sentence and the subject type right the predic the prediction label executive of the object of the sentence U.S Air Group okay um and the frequency of those objects so clearly something's going on here because these are blank and that's part probably blank because we didn't tell it to pull persons okay we told it to pull orgs so we could probably drop you these but see here where we have executive of so we have these um blanks because we didn't pull the executive's names I think it's what's going on now there's only a couple of them in here with the words acquires or sells so this isn't a great data set because it doesn't have enough examples to really make a cool graph um but that's why when we graph this you get this blank node right it's because there's a blank no connected to all of these but it's not all the same chairman if you had the names in there it would look a little better um that we can see that Transworld Airlines and U.S Air Group are related to each other because they have a connection forward and backwards and so one thing that I've figured out so right so we have org acquires transair group and org cells but it can't put both labels on there I think is why it freaks out and so I if we turn this off I think it wouldn't still run because it wouldn't allow us to have that same relationship in there twice but we could find a way to combine them together so I've acquired themselves or is bought by okay right so you have a couple of you could make it clear that you don't have a direction there to print this out a little better where the labels would appear on the lines themselves okay we'll see if we can figure that out in class but at least we got the relationship pulled out and you know how to get the data out of there so that you can use that maybe if you have some other software that you like a little better so on that note that is the end of the sectional information extraction thanks for tuning along and we will move on to our next section next week