Skip to content

Commit baa7067

Browse files
committed
[Build] Use C++17 Constructor for tiktoken.cpp when C++20 is unavailable
The basic_string_view constructor: ``` template< class It, class End > constexpr basic_string_view( It first, End last ); ``` requires C++20. To allow the code to compile with C++17, use the basic_string_view constructor: ``` constexpr basic_string_view( const CharT* s, size_type count ); ``` For #4661
1 parent 37cad01 commit baa7067

File tree

1 file changed

+10
-0
lines changed

1 file changed

+10
-0
lines changed

extension/llm/tokenizer/tiktoken.cpp

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,11 @@ Tiktoken::_split_with_allowed_special_token(
252252
return std::make_pair(std::nullopt, input);
253253
}
254254

255+
#if __cplusplus >= 202002L
255256
auto start = input.begin();
257+
#else
258+
const char* start = input.data();
259+
#endif
256260
std::string special;
257261
while (true) {
258262
if (!re2::RE2::FindAndConsume(&input, *_special_token_regex, &special)) {
@@ -262,9 +266,15 @@ Tiktoken::_split_with_allowed_special_token(
262266

263267
if (allowed_special.count(special) == 1) {
264268
// Found an allowed special token, split the text with it.
269+
#if __cplusplus >= 202002L
265270
return std::make_pair(
266271
special,
267272
re2::StringPiece(start, input.begin() - start - special.size()));
273+
#else
274+
return std::make_pair(
275+
special,
276+
re2::StringPiece(start, (input.data() - start) - special.size()));
277+
#endif
268278
} // else try to find the next special token
269279
}
270280

0 commit comments

Comments
 (0)