"detokenize" meaning in English

See detokenize in All languages combined, or Wiktionary

Verb

Forms: detokenizes [present, singular, third-person], detokenizing [participle, present], detokenized [participle, past], detokenized [past]
Etymology: de- + tokenize Etymology templates: {{prefix|en|de|tokenize}} de- + tokenize Head templates: {{en-verb}} detokenize (third-person singular simple present detokenizes, present participle detokenizing, simple past and past participle detokenized)
  1. (transitive, computing) To convert (a tokenized representation) back to the original form. Tags: transitive Categories (topical): Computing

Inflected forms

Download JSON data for detokenize meaning in English (1.6kB)

{
  "etymology_templates": [
    {
      "args": {
        "1": "en",
        "2": "de",
        "3": "tokenize"
      },
      "expansion": "de- + tokenize",
      "name": "prefix"
    }
  ],
  "etymology_text": "de- + tokenize",
  "forms": [
    {
      "form": "detokenizes",
      "tags": [
        "present",
        "singular",
        "third-person"
      ]
    },
    {
      "form": "detokenizing",
      "tags": [
        "participle",
        "present"
      ]
    },
    {
      "form": "detokenized",
      "tags": [
        "participle",
        "past"
      ]
    },
    {
      "form": "detokenized",
      "tags": [
        "past"
      ]
    }
  ],
  "head_templates": [
    {
      "args": {},
      "expansion": "detokenize (third-person singular simple present detokenizes, present participle detokenizing, simple past and past participle detokenized)",
      "name": "en-verb"
    }
  ],
  "lang": "English",
  "lang_code": "en",
  "pos": "verb",
  "senses": [
    {
      "categories": [
        {
          "kind": "other",
          "name": "English entries with incorrect language header",
          "parents": [
            "Entries with incorrect language header",
            "Entry maintenance"
          ],
          "source": "w"
        },
        {
          "kind": "other",
          "name": "English terms prefixed with de-",
          "parents": [],
          "source": "w"
        },
        {
          "kind": "topical",
          "langcode": "en",
          "name": "Computing",
          "orig": "en:Computing",
          "parents": [
            "Technology",
            "All topics",
            "Fundamental"
          ],
          "source": "w"
        }
      ],
      "glosses": [
        "To convert (a tokenized representation) back to the original form."
      ],
      "id": "en-detokenize-en-verb-RQOEY6wc",
      "links": [
        [
          "computing",
          "computing#Noun"
        ],
        [
          "convert",
          "convert"
        ],
        [
          "tokenize",
          "tokenize"
        ],
        [
          "representation",
          "representation"
        ]
      ],
      "raw_glosses": [
        "(transitive, computing) To convert (a tokenized representation) back to the original form."
      ],
      "tags": [
        "transitive"
      ],
      "topics": [
        "computing",
        "engineering",
        "mathematics",
        "natural-sciences",
        "physical-sciences",
        "sciences"
      ]
    }
  ],
  "word": "detokenize"
}
{
  "etymology_templates": [
    {
      "args": {
        "1": "en",
        "2": "de",
        "3": "tokenize"
      },
      "expansion": "de- + tokenize",
      "name": "prefix"
    }
  ],
  "etymology_text": "de- + tokenize",
  "forms": [
    {
      "form": "detokenizes",
      "tags": [
        "present",
        "singular",
        "third-person"
      ]
    },
    {
      "form": "detokenizing",
      "tags": [
        "participle",
        "present"
      ]
    },
    {
      "form": "detokenized",
      "tags": [
        "participle",
        "past"
      ]
    },
    {
      "form": "detokenized",
      "tags": [
        "past"
      ]
    }
  ],
  "head_templates": [
    {
      "args": {},
      "expansion": "detokenize (third-person singular simple present detokenizes, present participle detokenizing, simple past and past participle detokenized)",
      "name": "en-verb"
    }
  ],
  "lang": "English",
  "lang_code": "en",
  "pos": "verb",
  "senses": [
    {
      "categories": [
        "English entries with incorrect language header",
        "English lemmas",
        "English terms prefixed with de-",
        "English transitive verbs",
        "English verbs",
        "en:Computing"
      ],
      "glosses": [
        "To convert (a tokenized representation) back to the original form."
      ],
      "links": [
        [
          "computing",
          "computing#Noun"
        ],
        [
          "convert",
          "convert"
        ],
        [
          "tokenize",
          "tokenize"
        ],
        [
          "representation",
          "representation"
        ]
      ],
      "raw_glosses": [
        "(transitive, computing) To convert (a tokenized representation) back to the original form."
      ],
      "tags": [
        "transitive"
      ],
      "topics": [
        "computing",
        "engineering",
        "mathematics",
        "natural-sciences",
        "physical-sciences",
        "sciences"
      ]
    }
  ],
  "word": "detokenize"
}

This page is a part of the kaikki.org machine-readable English dictionary. This dictionary is based on structured data extracted on 2024-06-01 from the enwiktionary dump dated 2024-05-02 using wiktextract (384852d and db5a844). The data shown on this site has been post-processed and various details (e.g., extra categories) removed, some information disambiguated, and additional data merged from other sources. See the raw data download page for the unprocessed wiktextract data.

If you use this data in academic research, please cite Tatu Ylonen: Wiktextract: Wiktionary as Machine-Readable Structured Data, Proceedings of the 13th Conference on Language Resources and Evaluation (LREC), pp. 1317-1325, Marseille, 20-25 June 2022. Linking to the relevant page(s) under https://kaikki.org would also be greatly appreciated.