perl5241cdelta - what is new for cperl v5.24.1
This document describes perl-only differences between the cperl 5.24.0 release and the cperl 5.24.1 release.
If you are upgrading from an earlier release such as 5.22.3, first read perl5240cdelta, which describes differences between 5.22.3 and 5.24.0.
perl5.24.1 upstream was a security update to non-existing problem. cperl added PERLIO_DEBUG as optional handle to redirect DEBUGGING outout via -Dx switches to. This is only usable with DEBUGGING perls, which are not used in production, only during development. perl5.24.1 changed PERLIO_DEBUG to only be active for PerlIO debugging together with -Di.
@{ \327 \n } buffer overflowsFixed @{ \327 \n } tokenizer failures and heap buffer overflows in sv_vcatpvfn_flags() with wrong tracking of PL_linestr, the currently parsed line buffer. This can easily lead to security relevant exploits.
eval "q" . chr(overlarge) stack overflowIn eval "q" . chr(100000000064) generating the error message Can't find string terminator "XXX"' was overrunning a local buffer designed to hold a single utf8 char, since it wasn't allowing for the \0 at the end.
If the collisions for a hash key lookup exceeds 128 tries (i.e. a linear search in a linked list), this qualifies as a malicious hash DoS (Denial of Service) attack. Generally maximal 8 collisions appear in normal hash table usage. Every 8th such hash flood attack performs a sleep(2) to limit the impact.
This security scheme is much easier and faster than trying to hide the random hash seed with randomized iterators and collisions lists, which cperl doesn't use.
Detection of the destructive attack against Movable-Type, the third vector only, which tries to delete mt-config.cgi was added to was added to Storable 3.01c.
Warns with "SECURITY: Movable-Type CVE-2015-1592 Storable metasploit attack" but does not protect against it.
Detect the metasploit payload unix/reverse_perl and some existing variants. This is just a dumb match at startup against existing exploits in the wild, but not future variants. Warns with "SECURITY: metasploit reverse/bind shell payload", but do not protect against it. This warning is thrown even without -w.
Also detects the CVE-2012-1823 reverse/bind shell payload, which is widely exploited too. The security warning is called "SECURITY: CVE-2012-1823 reverse/bind shell payload".
Out-of-bounds check elimination in loops has been fixed for lexical counters. E.g. with my @a=(0..4); for my $i (0..$#a) { $a[$i] } each access to $a[$i] in the loop is now really converted to the unchecked faster aelem_u op.
Note that multideref ops are not yet converted to omit out-of-bounds checks. This is only implemented since cperl-5.25 (released with 5.26.0c), since it needs to widen the internal mderef structure.
Improvements when reading from arrays have been imported from perl 5.25. av_fetch() uses less branches reading from the end (negative indices), and a branch checking for freed @_ elements has been removed,
The new strEQc/strNEc macros are used instead of strEQ(s,"constant"). This enables word-wise comparison via memcpy, in opposite of byte-wise comparisons via strcmp with already known sizes. This is a 10% performance improvement under most optimization levels.
Use more strEQc, strNEc macros, when safe to use, i.e. the left buffer is big enough, now with Address Sanitizer fallbacks.
The new fast buffer comparison macros strEQc and strNEc compare a full string including the final \0, memEQc and memNEc just the start of a buffer, with constants strings. Note that valgrind and Address Sanitizer will complain about out of range access of the left side of the buffer. To access these buffers however is safe and will not lead to SIGBUS on stricter platforms. To prevent valgrind from warning on this, you may want to define -DVALGRIND, which uses a safe and slower fallback macro.
aassign: pre-allocate needed hash size with aassign, similar to arrays, avoiding run-time hash splits. e.g. my %h = (.. = .., .. => ..)>
This version is 30% faster overall in the Mail::SpamAssassin testsuite than cperl-5.24.0.
Pre-extend internal hashes and stashes to avoid unnecessary boot-time hash splits. %warnings::, %Config::, %utf8::, %version::.
New stringify_infnan(3) infnan_mode. Fix inf/nan detection on HP-UX and others. Use faster strEQc macros. Prefer memEQ for systems without memcmp, to use bcmp there. Add more expect_false() to inf/nan branches. Fix av and hv length types: protect from security sensitive overflows, add HVMAX_T and RITER_T Add new "Hash key too large" error. perl5 silently truncates it, but we prefer errors.
ReadKey.pm renamed to ReadKey_pm.PL, expand blockoptions specific variants already at installation, no load-time eval, demand-load Carp, remove unneeded AutoLoader, harmonize formatting
Allow a 2nd optional CV argument for B::OP::aux_list, fixing B::Deparse and thereby Data::Dumper and Test2 is_deeply.
protect sv in END during global destruction, esp. with B::C. fixes for missing . in @INC (cperl or -Dfortify_inc). Add i_netinet_in_systm. Removed i_netinet6_in6.
merged with upstream libyaml 0.1.7 avoid duplicate checks against NULL fix libyaml clang -Wlogical-op warnings fix libyaml clang -Wlogical-not-parentheses warnings
sum/min/max need to call SvGETMAGIC
set_subname memory fix by @bluhm from Sub::Name 0.20 [cpan #117072]
Fixes for older perls, esp. lexical $_ support.
Reinstate the &DB::sub setter, but no UTF8 support yet.
one more darwin fix for the wrong no library found warning for symlinked darwin libSystem.dylib libraries.
protect sv in END during global destruction, esp. with B::C. fixes for missing . in @INC (cperl or -Dfortify_inc).
From https://github.com/atoomic/IO-Socket-IP/:
- Support setting custom socket options with new Sockopts constructor parameter
- Restore blocking mode after ->connect errors [cpan #112334]
More Darwin thread fixes for clock_gettime, Sierra support, test improvements.
Merge cpan 2.024 with our 2.021_02, plus fix some problems in their new code.
Fixes for OpenBSD: Probe for netinet/in_systm.h Removed i_netinet6_in6 probe. This was never used due to a typo. It cannot be used due to RFC 2553.
no mathoms: call_sv instead of call_pv, get_cvs where available.
Better CopFILE_set, Fixup arenasize refcnt. Delay cvref to init2, properly set a SvRV to a XS sub. Optimize constpv for CvFILE (less constants to merge for gcc). Improve NV precision by one digit. Fix to compile in utf8_heavy.pl, abstract and set %INC. Fix generation of @B::C::Config::deps on Windows. Fix !C99 precedence bug (e.g. MSVC). Minor refactor to simplify save_hek. Use the new get_svs, get_avs, get_hvs macros. perlcc add --debug|-D Improve endav XSUB bump Abstract RITER_T and HVMAX_T for the various sizes, compat HEK_STATIC Defer REGCOMP for \P{} properties Change $sv->EXTFLAGS to compflags since 5.22 for CALLREGCOMP(). Turn off MGf_REFCOUNTED. global-buffer-overflow with dynamic COW strings, wrong savepvn args.
fixes for missing . in @INC (cperl or -Dfortify_inc).
The following additions or changes have been made to diagnostic output, including warnings and fatal error messages. For the complete list of diagnostic messages, see perldiag.
The Too many elements error is now triggered when accessing or extending an out of bounds array index or trying to insert too many hash keys. This is to prevent from silent hash or array overflows. Previously extending a hash beyond it's capable size was silently ignored, leading to performance degradation with overly high fill factors and extending an array failed only on memory exhaustion, but the signed index led to an index overflow between I32 and U32, resp. I64 and U64.
Even worse, accessing overflown unsigned array indices would silently access the signed counterpart, indices at the end.
Note that the out of bound error message with shaped arrays is different. (cperl only)
The Panic: hash key too long error is now thrown with overlarge hash keys in every hv_common access and in Cpanel::JSON::XS. perl5 still silently ignores those failures, and truncates the keys. Many more similar panic: (file|keyword|mro|stash)? name too long errors were added to the parser and compiler to protect from overlong names (> I32_MAX, 2147483647). (cperl only)
The new security warning "Hash flood" was added. See "Protect and warn on hash flood DoS". (cperl only)
The new security warnings "metasploit reverse/bind shell payload" and "CVE-2012-1823 reverse/bind shell payload" were added, detecting the existing metasploit/libxploit and phpcgi CVE-2012-1823 reverse and bind shells. See "Warn on metasploit reverse shells" (cperl only)
Added sanitize_address config entry and probe, and matching USE_SANITIZE_ADDRESS config.h define. (cperl only)
Removed i_netinet6_in6 Config entry and probe, and matching I_NETINET6_I6 config.h define, which was a typo. This was added with cperl-5.22.2 and was never used due to the typo. It cannot be used due to RFC 2553. (cperl only)
Added i_netinet_in_systm probe. (cperl only)
make minitest has been vastly improved
Fix tests for the optional -DNODEFAULT_SHAREKEYS configuration.
Add hash flood tests with pre-created collision data for 511 collisions on seed=0.
Split t/op/tr.t into unicode and ebcdic variant
Don't warn on missing Pod::Man in CORE tests. Pod::Man is not a dependency for EUMM, as it needs EUMM at first.
Increasing the hash size to values larger then U32_MAX are now silently ignored. This only affects 64-bit platforms. Normally a hash split to an overlarge value should die, but historically it silently ignored the split, it just increases the fill rate and decreases performance.
Fixed several heap-buffer-overflows detected by asan: use-after-free in Dynaloader (ReadKey probe with -DDEBUG_LEAKING_SCALAR), heap-overflow in gv_fetchfile (t/comp/parser.t), heap-overflow with signatures, heap-overflow in XSLoader, invalid memEQc in toke.c missing out on parsing #!perl -opts, B-C global-buffer-overflow with dynamic COW strings, wrong savepvn args.
There are still heap-use-after-free problems with perlcc and PERL_DESTRUCT_LEVEL=2.
See [cperl #207]
Fixed overwriting the HVhek_UNSHARED bit in the hash loop broken with v5.9.
This fixed -DNODEFAULT_SHAREKEYS. In the default configuration without NODEFAULT_SHAREKEYS since 5.003_001 all hash keys are stored twice, once in the hash and once again in PL_strtab, the global string table, with the benefit of faster hash loops and copies. Almost all hashtables get the SHAREKEYS bit. With -Accflags=-DNODEFAULT_SHAREKEYS simple scripts are 20-30% faster. [cperl #201]
Fix HEK_TAINTED check for HEf_SVKEY values. A HEf_SVKEY hek has no tainted flag, the pointed to SV has. This is a cperl-only security feature.
Only clear LS_COLORS for glob
When miniperl calls csh to implement glob(), we cleared %ENV temporarily to avoid csh dying on invalid values for things like LS_COLORS. That has proven to have far too many problems, since many system-dependent env vars are necessary for calling an external process. See the [perl #126041] ticket for details.
A better solution is temporarily to clear only those vars that are known to be problematic and make csh possibly fail. There only hap- pens to be one of those at present, namely LS_COLORS.
A SEGV in mess_sv during global destruction with a DEBUGGING perl and -DS been fixed, occuring when we wanted to report the location of an error when curcop has already been freed.
Testcase: ./miniperl -DS -e '$_="f"; s/./"&".$&/ee'
A SEGV in ck_shift with an empty/wrong current function, caused by a syntax error has been fixed. The syntax error is now reported lateron. Testcase: 'qq{@{sub{q}}]]}}; s0{shift'
Since Perl 5.20, line numbers have been off by one when perl is invoked with the -x switch. This has been fixed. [perl #128508]
Handle missing Unicode heredoc terminators correctly. E.g. perl -CS -e 'use utf8; q«' prints now Can't find string terminator "«" anywhere before EOF at -e line 1.
Mentioning a constant twice in a row does not lead to assertions errors with DEBUGGING builds, such as e.g. sub ub(){0} ub ub.
Fixed overwriting the HVhek_UNSHARED bit in the hash loop broken with v5.9.
This fixed -DNODEFAULT_SHAREKEYS. In the default configuration without NODEFAULT_SHAREKEYS since 5.003_001 all hash keys are stored twice, once in the hash and once again in PL_strtab, the global string table, with the benefit of faster hash loops and copies. Almost all hashtables get the SHAREKEYS bit. (#201)
Array and hash index overflow are now properly detected and throw an "Too many elements" error.
E.g. on 32bit $ary[2147483648] will lead to a compile-time error, $i=2147483648; $ary[$i] to a run-time error. Before 5.24c or with perl5 those two would silently overflow to -1, i.e. accessing the last element.
When inserting more than U32, i.e. 4294967295 hash elements, the same error is now thrown. Before 5.24c or with perl5 the element would have been inserted, with 32bit the hash table would not have been extended and the load factor would increase, the collisions and performance would decrease. On 64bit the hash table would have beed increased, but the elements would collide on the last element, leading to the same problems, just more dramatic.
cperl 5.24.1 represents approximately 3 months of development since cperl 5.24.0c and contains approximately 15,000 lines of changes across 250 files from 11 authors.
Excluding auto-generated files, documentation and release tools, there were approximately 4,900 lines of changes to 130 .pm, .t, .c and .h files.
Perl continues to flourish into its third decade thanks to a vibrant community of users and developers. The following people are known to have contributed the improvements that became cperl 5.24.1c:
Reini Urban, Father Chrysostomos, David Mitchell, Daniel Dragan, Tony Cook, Lukas Mai, Yves Orton, Aristotle Pagaltzis, Misty De Meo, Karl Williamson, James Raspass, Nicholas Rochelemange.
The list above is almost certainly incomplete as it is automatically generated from version control history. In particular, it does not include the names of the (very much appreciated) contributors who reported issues to the cperl issue tracker.
Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.
For a more complete list of all of Perl's historical contributors, please see the AUTHORS file in the Perl source distribution.
If you find what you think is a bug, you might check the articles recently posted to the comp.lang.perl.misc newsgroup and the perl bug database at https://rt.perl.org/ . There may also be information at http://www.perl.org/ , the Perl Home Page.
If you believe you have an unreported bug, please run the cperlbug program included with your release. Be sure to trim your bug down to a tiny but sufficient test case. Your bug report, along with the output of perl -V, will be sent off to perlbug@perl.org to be analysed by the Perl porting team.
If you think it's a cperl specific bug or trust the cperl developers more please file an issue at https://github.com/perl11/cperl/issues.
If the bug you are reporting has security implications which make it inappropriate to send to a publicly archived mailing list, then see "SECURITY VULNERABILITY CONTACT INFORMATION" in perlsec for details of how to report the issue.
The Changes file for an explanation of how to view exhaustive details on what changed.
The INSTALL file for how to build Perl.
The README file for general stuff.
The Artistic and Copying files for copyright information.