See tokenizer in All languages combined, or Wiktionary
Download JSONL data for tokenizer meaning in Anglais (1.5kB)
{
"categories": [
{
"kind": "other",
"name": "Lemmes en anglais",
"parents": [],
"source": "w"
},
{
"kind": "other",
"name": "Noms communs en anglais",
"parents": [],
"source": "w"
},
{
"kind": "other",
"name": "Wiktionnaire:Étymologies manquantes en anglais",
"parents": [],
"source": "w"
},
{
"kind": "other",
"name": "Anglais",
"orig": "anglais",
"parents": [],
"source": "w"
}
],
"etymology_texts": [
"Étymologie manquante ou incomplète. Si vous la connaissez, vous pouvez l’ajouter en cliquant ici."
],
"forms": [
{
"form": "tokenizers",
"ipas": [
"\\toʊ.kə.naɪ.zəɹz\\"
],
"tags": [
"plural"
]
}
],
"lang": "Anglais",
"lang_code": "en",
"pos": "noun",
"pos_title": "Nom commun",
"related": [
{
"word": "tokeniser"
},
{
"word": "tokenization"
},
{
"word": "tokenize"
}
],
"senses": [
{
"categories": [
{
"kind": "other",
"name": "Exemples en anglais",
"parents": [],
"source": "w"
},
{
"kind": "other",
"name": "Exemples en anglais à traduire",
"parents": [],
"source": "w"
},
{
"kind": "other",
"name": "Lexique en anglais de l’informatique",
"parents": [],
"source": "w"
}
],
"examples": [
{
"ref": "Syntactic Wordclass Tagging - Page 121, H. van Halteren - 1999",
"text": "In the Unix-based world, there are two general tools which allow a user to write a natural language tokenizer: Lex (chap. 3 of Aho et al. 1986) and Awk (Aho 1988)."
}
],
"glosses": [
"Parseur en tokens. Par exemple cela permet de transformer un texte en plusieurs mots séparés par des espaces."
],
"id": "fr-tokenizer-en-noun-Jstjla2W",
"topics": [
"computing"
]
}
],
"word": "tokenizer"
}
{
"categories": [
"Lemmes en anglais",
"Noms communs en anglais",
"Wiktionnaire:Étymologies manquantes en anglais",
"anglais"
],
"etymology_texts": [
"Étymologie manquante ou incomplète. Si vous la connaissez, vous pouvez l’ajouter en cliquant ici."
],
"forms": [
{
"form": "tokenizers",
"ipas": [
"\\toʊ.kə.naɪ.zəɹz\\"
],
"tags": [
"plural"
]
}
],
"lang": "Anglais",
"lang_code": "en",
"pos": "noun",
"pos_title": "Nom commun",
"related": [
{
"word": "tokeniser"
},
{
"word": "tokenization"
},
{
"word": "tokenize"
}
],
"senses": [
{
"categories": [
"Exemples en anglais",
"Exemples en anglais à traduire",
"Lexique en anglais de l’informatique"
],
"examples": [
{
"ref": "Syntactic Wordclass Tagging - Page 121, H. van Halteren - 1999",
"text": "In the Unix-based world, there are two general tools which allow a user to write a natural language tokenizer: Lex (chap. 3 of Aho et al. 1986) and Awk (Aho 1988)."
}
],
"glosses": [
"Parseur en tokens. Par exemple cela permet de transformer un texte en plusieurs mots séparés par des espaces."
],
"topics": [
"computing"
]
}
],
"word": "tokenizer"
}
This page is a part of the kaikki.org machine-readable Anglais dictionary. This dictionary is based on structured data extracted on 2024-07-10 from the frwiktionary dump dated 2024-07-01 using wiktextract (6dfc2a1 and 7cfad79). The data shown on this site has been post-processed and various details (e.g., extra categories) removed, some information disambiguated, and additional data merged from other sources. See the raw data download page for the unprocessed wiktextract data.
If you use this data in academic research, please cite Tatu Ylonen: Wiktextract: Wiktionary as Machine-Readable Structured Data, Proceedings of the 13th Conference on Language Resources and Evaluation (LREC), pp. 1317-1325, Marseille, 20-25 June 2022. Linking to the relevant page(s) under https://kaikki.org would also be greatly appreciated.