{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "<a href='http://www.pieriandata.com'> <img src='../Pierian_Data_Logo.png' /></a>\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition (NER)\n",
    "spaCy has an **'ner'** pipeline component that identifies token spans fitting a predetermined set of named entities. These are available as the `ents` property of a `Doc` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform standard imports\n",
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a function to display basic entity info:\n",
    "def show_ents(doc):\n",
    "    if doc.ents:\n",
    "        for ent in doc.ents:\n",
    "            print(ent.text+' - '+ent.label_+' - '+str(spacy.explain(ent.label_)))\n",
    "    else:\n",
    "        print('No named entities found.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Washington, DC - GPE - Countries, cities, states\n",
      "next May - DATE - Absolute or relative dates or periods\n",
      "the Washington Monument - ORG - Companies, agencies, institutions, etc.\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(u'May I go to Washington, DC next May to see the Washington Monument?')\n",
    "\n",
    "show_ents(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see tokens combine to form the entities `Washington, DC`, `next May` and `the Washington Monument`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Entity annotations\n",
    "`Doc.ents` are token spans with their own set of annotations.\n",
    "<table>\n",
    "<tr><td>`ent.text`</td><td>The original entity text</td></tr>\n",
    "<tr><td>`ent.label`</td><td>The entity type's hash value</td></tr>\n",
    "<tr><td>`ent.label_`</td><td>The entity type's string description</td></tr>\n",
    "<tr><td>`ent.start`</td><td>The token span's *start* index position in the Doc</td></tr>\n",
    "<tr><td>`ent.end`</td><td>The token span's *stop* index position in the Doc</td></tr>\n",
    "<tr><td>`ent.start_char`</td><td>The entity text's *start* index position in the Doc</td></tr>\n",
    "<tr><td>`ent.end_char`</td><td>The entity text's *stop* index position in the Doc</td></tr>\n",
    "</table>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "500 dollars 4 6 20 31 MONEY\n",
      "Microsoft 11 12 53 62 ORG\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(u'Can I please borrow 500 dollars from you to buy some Microsoft stock?')\n",
    "\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.start, ent.end, ent.start_char, ent.end_char, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NER Tags\n",
    "Tags are accessible through the `.label_` property of an entity.\n",
    "<table>\n",
    "<tr><th>TYPE</th><th>DESCRIPTION</th><th>EXAMPLE</th></tr>\n",
    "<tr><td>`PERSON`</td><td>People, including fictional.</td><td>*Fred Flintstone*</td></tr>\n",
    "<tr><td>`NORP`</td><td>Nationalities or religious or political groups.</td><td>*The Republican Party*</td></tr>\n",
    "<tr><td>`FAC`</td><td>Buildings, airports, highways, bridges, etc.</td><td>*Logan International Airport, The Golden Gate*</td></tr>\n",
    "<tr><td>`ORG`</td><td>Companies, agencies, institutions, etc.</td><td>*Microsoft, FBI, MIT*</td></tr>\n",
    "<tr><td>`GPE`</td><td>Countries, cities, states.</td><td>*France, UAR, Chicago, Idaho*</td></tr>\n",
    "<tr><td>`LOC`</td><td>Non-GPE locations, mountain ranges, bodies of water.</td><td>*Europe, Nile River, Midwest*</td></tr>\n",
    "<tr><td>`PRODUCT`</td><td>Objects, vehicles, foods, etc. (Not services.)</td><td>*Formula 1*</td></tr>\n",
    "<tr><td>`EVENT`</td><td>Named hurricanes, battles, wars, sports events, etc.</td><td>*Olympic Games*</td></tr>\n",
    "<tr><td>`WORK_OF_ART`</td><td>Titles of books, songs, etc.</td><td>*The Mona Lisa*</td></tr>\n",
    "<tr><td>`LAW`</td><td>Named documents made into laws.</td><td>*Roe v. Wade*</td></tr>\n",
    "<tr><td>`LANGUAGE`</td><td>Any named language.</td><td>*English*</td></tr>\n",
    "<tr><td>`DATE`</td><td>Absolute or relative dates or periods.</td><td>*20 July 1969*</td></tr>\n",
    "<tr><td>`TIME`</td><td>Times smaller than a day.</td><td>*Four hours*</td></tr>\n",
    "<tr><td>`PERCENT`</td><td>Percentage, including \"%\".</td><td>*Eighty percent*</td></tr>\n",
    "<tr><td>`MONEY`</td><td>Monetary values, including unit.</td><td>*Twenty Cents*</td></tr>\n",
    "<tr><td>`QUANTITY`</td><td>Measurements, as of weight or distance.</td><td>*Several kilometers, 55kg*</td></tr>\n",
    "<tr><td>`ORDINAL`</td><td>\"first\", \"second\", etc.</td><td>*9th, Ninth*</td></tr>\n",
    "<tr><td>`CARDINAL`</td><td>Numerals that do not fall under another type.</td><td>*2, Two, Fifty-two*</td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Adding a Named Entity to a Span\n",
    "Normally we would have spaCy build a library of named entities by training it on several samples of text.<br>In this case, we only want to add one value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "U.K. - GPE - Countries, cities, states\n",
      "$6 million - MONEY - Monetary values, including unit\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(u'Tesla to build a U.K. factory for $6 million')\n",
    "\n",
    "show_ents(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=green>Right now, spaCy does not recognize \"Tesla\" as a company.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.tokens import Span\n",
    "\n",
    "# Get the hash value of the ORG entity label\n",
    "ORG = doc.vocab.strings[u'ORG']  \n",
    "\n",
    "# Create a Span for the new entity\n",
    "new_ent = Span(doc, 0, 1, label=ORG)\n",
    "\n",
    "# Add the entity to the existing Doc object\n",
    "doc.ents = list(doc.ents) + [new_ent]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=green>In the code above, the arguments passed to `Span()` are:</font>\n",
    "-  `doc` - the name of the Doc object\n",
    "-  `0` - the *start* index position of the span\n",
    "-  `1` - the *stop* index position (exclusive)\n",
    "-  `label=ORG` - the label assigned to our entity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tesla - ORG - Companies, agencies, institutions, etc.\n",
      "U.K. - GPE - Countries, cities, states\n",
      "$6 million - MONEY - Monetary values, including unit\n"
     ]
    }
   ],
   "source": [
    "show_ents(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Adding Named Entities to All Matching Spans\n",
    "What if we want to tag *all* occurrences of \"Tesla\"? In this section we show how to use the PhraseMatcher to identify a series of spans in the Doc:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first - ORDINAL - \"first\", \"second\", etc.\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(u'Our company plans to introduce a new vacuum cleaner. '\n",
    "          u'If successful, the vacuum cleaner will be our first product.')\n",
    "\n",
    "show_ents(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import PhraseMatcher and create a matcher object:\n",
    "from spacy.matcher import PhraseMatcher\n",
    "matcher = PhraseMatcher(nlp.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the desired phrase patterns:\n",
    "phrase_list = ['vacuum cleaner', 'vacuum-cleaner']\n",
    "phrase_patterns = [nlp(text) for text in phrase_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(2689272359382549672, 7, 9), (2689272359382549672, 14, 16)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply the patterns to our matcher object:\n",
    "matcher.add('newproduct', None, *phrase_patterns)\n",
    "\n",
    "# Apply the matcher to our Doc object:\n",
    "matches = matcher(doc)\n",
    "\n",
    "# See what matches occur:\n",
    "matches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we create Spans from each match, and create named entities from them:\n",
    "from spacy.tokens import Span\n",
    "\n",
    "PROD = doc.vocab.strings[u'PRODUCT']\n",
    "\n",
    "new_ents = [Span(doc, match[1],match[2],label=PROD) for match in matches]\n",
    "\n",
    "doc.ents = list(doc.ents) + new_ents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vacuum cleaner - PRODUCT - Objects, vehicles, foods, etc. (not services)\n",
      "vacuum cleaner - PRODUCT - Objects, vehicles, foods, etc. (not services)\n",
      "first - ORDINAL - \"first\", \"second\", etc.\n"
     ]
    }
   ],
   "source": [
    "show_ents(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Counting Entities\n",
    "While spaCy may not have a built-in tool for counting entities, we can pass a conditional statement into a list comprehension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29.50 - MONEY - Monetary values, including unit\n",
      "five dollars - MONEY - Monetary values, including unit\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(u'Originally priced at $29.50, the sweater was marked down to five dollars.')\n",
    "\n",
    "show_ents(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([ent for ent in doc.ents if ent.label_=='MONEY'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=blue>Problem with Line Breaks</font>\n",
    "\n",
    "<div class=\"alert alert-info\" style=\"margin: 20px\">There's a <a href='https://github.com/explosion/spaCy/issues/1717'>known issue</a> with <strong>spaCy v2.0.12</strong> where some linebreaks are interpreted as `GPE` entities:</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.12'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29.50 - MONEY - Monetary values, including unit\n",
      "\n",
      " - GPE - Countries, cities, states\n",
      "five dollars - MONEY - Monetary values, including unit\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(u'Originally priced at $29.50,\\nthe sweater was marked down to five dollars.')\n",
    "\n",
    "show_ents(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=blue>However, there is a simple fix that can be added to the nlp pipeline:</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quick function to remove ents formed on whitespace:\n",
    "def remove_whitespace_entities(doc):\n",
    "    doc.ents = [e for e in doc.ents if not e.text.isspace()]\n",
    "    return doc\n",
    "\n",
    "# Insert this into the pipeline AFTER the ner component:\n",
    "nlp.add_pipe(remove_whitespace_entities, after='ner')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29.50 - MONEY - Monetary values, including unit\n",
      "five dollars - MONEY - Monetary values, including unit\n"
     ]
    }
   ],
   "source": [
    "# Rerun nlp on the text above, and show ents:\n",
    "doc = nlp(u'Originally priced at $29.50,\\nthe sweater was marked down to five dollars.')\n",
    "\n",
    "show_ents(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more on **Named Entity Recognition** visit https://spacy.io/usage/linguistic-features#101"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Noun Chunks\n",
    "`Doc.noun_chunks` are *base noun phrases*: token spans that include the noun and words describing the noun. Noun chunks cannot be nested, cannot overlap, and do not involve prepositional phrases or relative clauses.<br>\n",
    "Where `Doc.ents` rely on the **ner** pipeline component, `Doc.noun_chunks` are provided by the **parser**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `noun_chunks` components:\n",
    "<table>\n",
    "<tr><td>`.text`</td><td>The original noun chunk text.</td></tr>\n",
    "<tr><td>`.root.text`</td><td>The original text of the word connecting the noun chunk to the rest of the parse.</td></tr>\n",
    "<tr><td>`.root.dep_`</td><td>Dependency relation connecting the root to its head.</td></tr>\n",
    "<tr><td>`.root.head.text`</td><td>The text of the root token's head.</td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Autonomous cars - cars - nsubj - shift\n",
      "insurance liability - liability - dobj - shift\n",
      "manufacturers - manufacturers - pobj - toward\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(u\"Autonomous cars shift insurance liability toward manufacturers.\")\n",
    "\n",
    "for chunk in doc.noun_chunks:\n",
    "    print(chunk.text+' - '+chunk.root.text+' - '+chunk.root.dep_+' - '+chunk.root.head.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Doc.noun_chunks` is a  generator function\n",
    "Previously we mentioned that `Doc` objects do not retain a list of sentences, but they're available through the `Doc.sents` generator.<br>It's the same with `Doc.noun_chunks` - lists can be created if needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "object of type 'generator' has no len()",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-21-8b52b37c204e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdoc\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnoun_chunks\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m: object of type 'generator' has no len()"
     ]
    }
   ],
   "source": [
    "len(doc.noun_chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list(doc.noun_chunks))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more on **noun_chunks** visit https://spacy.io/usage/linguistic-features#noun-chunks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Now you should be more familiar with both named entities and noun chunks. In the next section we revisit the NER visualizer.\n",
    "## Next up: Visualizing NER"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
