Split with matches

[Keats]

I was trying to implement http://jinja.pocoo.org/docs/dev/templates/#title and I was surprised to realise that matches were not included on split like they are in js or python.
In JS:

"foo bar".split(/(\s+)/)
["foo", " ", "bar"]

Is it possible to get a split_including_matches that matches that behaviour?
My naive implementation of the filter was:

static ref WORD_BEGINNING_SPLIT_RE: Regex = Regex::new(r"([-\s\(\{\[<]+)").unwrap();

&WORD_BEGINNING_SPLIT_RE
        .split(&s)
        .map(|item| {
            let mut c = item.chars();
            match c.next() {
                None => "".to_string(),
                Some(f) => f.to_uppercase().collect::<String>() + &c.as_str().to_lowercase()
            }
        })
        .collect::<Vec<_>>()
        .join("")

but since it doesn't include the matches, if I join at the end I will have lost all the whitespaces.

The current implementation takes 50 lines to do that https://github.com/Keats/tera/blob/master/src/filters/string.rs#L105-L155

[BurntSushi]

I'm not sure I follow. Could you be more explicit with an example that includes your expected output?

As for Python, it looks like its behavior matches this library:

>>> import re
>>> re.split('\s+', 'this  is a \n\ntest')
['this', 'is', 'a', 'test']

Javascript also works similarly:

"this   is a \n\ntest".split(/\s+/)
["this", "is", "a", "test"]

Hmm... You're using a capturing group, I wonder if that plays a role:

"this   is a \n\ntest".split(/(\s+)/)
["this", "   ", "is", " ", "a", " ↵↵", "test"]

Ah ha! Same for Python too:

>>> import re
>>> re.split('(\s+)', 'this  is a \n\ntest')
['this', '  ', 'is', ' ', 'a', ' \n\n', 'test']

The key part of the Python docs is:

If capturing parentheses are used in pattern, then the text of all groups in the pattern are also returned as part of the resulting list.

This seems a little strange to me, and honestly, I haven't read through your particular problem closely. I should have time for a more careful review later today and hopefully come up with a better answer.

[BurntSushi]

@Keats Here's how I'd do it:

#[macro_use]
extern crate lazy_static;
extern crate regex;

use regex::Regex;

pub fn title(s: &str) -> String {
    lazy_static! {
        static ref WORDS: Regex = Regex::new(
            r"\b(?P<first>\w)(?P<rest>\w*)\b"
        ).unwrap();
    };
    WORDS.replace_all(s, |caps: &regex::Captures| {
        let first = caps["first"].to_uppercase();
        let rest = caps["rest"].to_lowercase();
        format!("{}{}", first, rest)
    })
}

#[cfg(test)]
mod tests {
    use super::title;

    #[test]
    fn it_works() {
        assert_eq!("F", &*title("f"));
        assert_eq!("Foo Bar", &*title("foo bar"));
    }
}

Perhaps this isn't accounting for something that your implementation handles?

(I think this might be an example of an XY problem. The semantics your requesting in split seem awfully confusing to me.)

[Keats]

I agree the current code is confusing, hence why i was looking to improve it.
title should work on strings with n words, "foo bar" was an example. Taking from the testcases:

            // input       output
            ("foo bar", "Foo Bar"),
            ("foo\tbar", "Foo\tBar"),
            ("foo  bar", "Foo  Bar"),
            ("f bar f", "F Bar F"),
            ("foo-bar", "Foo-Bar"),
            ("FOO\tBAR", "Foo\tBar"),
            ("foo (bar)", "Foo (Bar)"),
            ("foo (bar) ", "Foo (Bar) "),
            ("foo {bar}", "Foo {Bar}"),
            ("foo [bar]", "Foo [Bar]"),
            ("foo <bar>", "Foo <Bar>"),
            ("  foo  bar", "  Foo  Bar"),
            ("\tfoo\tbar\t", "\tFoo\tBar\t"),
            ("foo bar ", "Foo Bar "),
            ("foo bar\t", "Foo Bar\t")

The Jinja2 implementation is very straigthforward because of the way split with captures works in python:

def do_title(s):
    """Return a titlecased version of the value. I.e. words will start with
    uppercase letters, all remaining characters are lowercase.
    """
    return ''.join(
        [item[0].upper() + item[1:].lower()
         for item in _word_beginning_split_re.split(soft_unicode(s))
         if item])

From: https://github.com/pallets/jinja/blob/master/jinja2/filters.py#L183-L190

Time to sleep right now but i'll answer tomorrow if you have more questions

[BurntSushi]

My implementation passes your test quite:

    #[test]
    fn it_works() {
        let cases = &[
            ("foo bar", "Foo Bar"),
            ("foo\tbar", "Foo\tBar"),
            ("foo  bar", "Foo  Bar"),
            ("f bar f", "F Bar F"),
            ("foo-bar", "Foo-Bar"),
            ("FOO\tBAR", "Foo\tBar"),
            ("foo (bar)", "Foo (Bar)"),
            ("foo (bar) ", "Foo (Bar) "),
            ("foo {bar}", "Foo {Bar}"),
            ("foo [bar]", "Foo [Bar]"),
            ("foo <bar>", "Foo <Bar>"),
            ("  foo  bar", "  Foo  Bar"),
            ("\tfoo\tbar\t", "\tFoo\tBar\t"),
            ("foo bar ", "Foo Bar "),
            ("foo bar\t", "Foo Bar\t")
        ];
        for &(given, expected) in cases {
            assert_eq!(expected, &*title(given));
        }
    }

I'd argue it's even more straight-forward than jinja2's, because it doesn't rely on a subtle semantic of re.split. ;-)

[BurntSushi]

Also, the regex jinga2 uses is: _word_beginning_split_re = re.compile(r'([-\s\(\{\[\<]+)(?u)'). This will actually miss a number of cases that my implementation will catch, since \b is a Unicode-aware word boundary. jinga2 hard-codes their own definition of word boundary.

[Keats]

I shouldn't do programming at 1am, I completely misread your regex yesterday.
It works perfectly, thanks!

[yangsibai]

Sorry to add my comments at a closed issue.

I have the same problem like Keats'.

I want to use the split method and keep groups in the results.

For example:

In python:

>>> import re;
>>> re.split("(x)", "1x2");
['1', 'x', '2']

In rust:

use regex::Regex;
let re = Regex::new(r"x").unwrap();
println!("{:?}", re.split("1x2").collect::<Vec<&str>>()); // the result is `["1", "2"]`, x is missing

How to do this in regex crate? Thank you.

[BurntSushi]

@yangsibai Please open a new issue and describe the problem you're trying to solve.

[BurntSushi]

By "describe the problem you're trying to solve" I mean "give a description of your problem that doesn't mention Python's regex library or Rust's regex library."

[yangsibai]

@BurntSushi Sorry. I have opened a new issue #330 .